AI 模型和平台
OpenAI 为AI培训首选荣誉共享机器人.txt信号
在 Google 上的首选来源中添加 Unite.AI
OpenAI 宣布于2026年10月30日表示将尊重机器人.txt中的共享信号，文件网站用来表达对其内容使用方式的偏好，因此站点运营商可以同时向多个参与供应商传达单一AI训练偏好。
横跨供应商的自愿共享信号
OpenAI在公告中称这项努力是自愿的，称解决了现有的负担：直到现在，网站运营商必须确定个人爬行者，并为每个供应商分别维持规则。 一个共享的信号，称公司让出版商和网站运营商同时向多个参与供应商传达同样的偏好。 OpenAI 称旨在在未来几个月内实施信号，并将与执行细节分享另一个更新。
称公司的新信号不会取代它已经通过机器人提供的控制.txt，称将继续应用并被尊重。 OpenAI将此举设定为借鉴了互联网工程任务组AI优惠工作组正在进行的工作，称正与其他工作组参与者一起发布公告，并鼓励其他AI供应商尊重该信号。 称将继续随着标准对话的发展而调整其做法。
信号是如何工作的
由Anthropic的Kevin Kelley撰写，日期为2026年10月30日，至2027年5月3日。 IETF的数据跟踪器指出表示个人的Internet-草案没有得到IETF的认可，在IETF标准进程中没有正式的地位。
在草案下，一个服务运营商通过在用户-Agent行中以每个目标的政策符号键入的写出普通机器人.txt组来约束使用偏好，如“不进行AI训练”。 推出草案没有新的指令名称，也没有新的语法：路径语言，最长的相匹配优先级，以及作为RFC 9309发布的机器人.txt标准的分组结构不变。 因此，通过互联网部署的现有解析器称草案将像以前一样继续解释交通管理信号，并将忽略政策数据组，因为任何组别都给一个他们不认识的用户取名agent。
草案定义了交叉复仇政策符号 usage ai-training. 其实例组将“用户-Agent: usage ai-training”与“Disallow: /”和“Allow:/press-kit/”两行对齐，表示倾向于对所有路径进行人工智能培训，但/press-kit/下的道路除外。 Wildcard groups，意为"User-Agent: *"，不得被解释为与政策符号相匹配，如果一个文件中没有组包含该符号，那么就没有表示任何偏好。
草案还称，政策标志不是爬行行为的指令。 内容-使用偏好和爬行器交通管理信号是独立的和正交的，是称，私人协议可能压倒通过机制表达的偏好。 它所宣称的目标包括路径特定约束，距离RFC 9309的最小距离，没有大规模地重复规则，不执行机制的解说者行为不变，以及消费者之间一致的解释。
信号后面的词汇
这个标志的意义追踪着工作组词汇草稿，即 " 表示AI使用优惠的词汇表 " (草案-itf-aiprif-vocab-08)，由开放未来的Paul Keller和Mozilla的Martin Thomson编写，最近一次更新是在2026年9月，作为工作组文件，旨在作为拟议标准。 该词汇定义了AI Training是使用一种资产来修改基因类AI模型的所学参数，排除了完全用于非基因任务的模型的培训，它定义了基因类AI模型是用于或提供使用的一种或多种方式生成合成内容的一种。
词汇还定义了一个单独的AI Use类，涵盖作为输入基因AI模型而不是由用户直接提供的资产，以及一个主要目的是选择资产并引导用户前往其所在地的应用程序的搜索类。 每个类别都得到允许的、不允许的或未知的优惠价值，在声明相冲突时，适用最限制性的优惠。 词汇草案指出，它没有确保偏好得到遵循，而是将是否遵循的选择留给接受者，给读者的说明提醒说，其内容尚未反映工作组的共识。 其中一节仍被跟踪列为一个未决问题。 一节指出，遵守培训偏好包括传达与培训资产有关的偏好，以及分发时的模式。
Suresh Krishnan of Cisco，词汇编辑汤姆森和凯勒，他们还列出了个人参与者，仅因IETF参与者作为个人贡献而注明其隶属关系，这些组织包括OpenAI,Google,Microsoft,Meta,Cloudflare，共同爬行基金会，新闻公司，"金融时报",BBC，电子前沿基金会，以及创意共同公司等组织。
OpenAI 现有的爬行控制
OpenAI的爬行者文件(称公司将保留在新信号旁边)将GPTBot描述为agent，它爬行了可用于训练其基因化AI基础模型的内容；不批准GPTBot表示一个站点的内容不应用于这种训练。 审调局-SearchBot在ChatGPT的搜索功能中显示网站，选择退出的网站将不会在ChatGPT搜索答案中显示，尽管它们仍然可以作为导航链接出现。 审调局-AdsBot验证作为ChatGPT广告提交的网页的安全性，它所收集的数据不用于培训基因AI基础模型。 ChatGPT-User处理由ChatGPT用户动作所触发的页面访问，而不是自动爬行，文档指出机器人.txt规则可能不适用于这些访问。
根据文件，每个设置都独立于其他设置，因此网站管理员可以允许审调处-SearchBot在搜索结果中出现，同时不允许GPTBot进行培训。 搜索调整可能需要在网站的机器人.txt更新后24小时左右。 OpenAI 称将在未来几个月执行共享信号时共享执行细节。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
AI Models & Platforms
OpenAI to Honor Shared Robots.txt Signal for AI Training Preferences
Add Unite.AI to your preferred sources on Google
OpenAI announced on October 30, 2026 that it will honor a shared signal in robots.txt, the file websites use to express preferences about how their content is used, so that site operators can communicate a single AI training preference to multiple participating providers at once.
A Voluntary Signal Shared Across Providers
In its announcement, OpenAI described the effort as voluntary and said it addresses an existing burden: until now, website operators had to identify individual crawlers and maintain separate rules for each and every provider. A shared signal, the company said, lets publishers and website operators communicate the same preference to multiple participating providers at once. OpenAI said it aims to implement the signal in the coming months and will share another update with implementation details when it does.
The company said the new signal will not replace the controls it already provides through robots.txt, which it said will continue to apply and be respected. OpenAI framed the move as drawing on the ongoing work of the Internet Engineering Task Force’s AI Preferences working group, said it is joining other working group participants in making the announcement, and encouraged other AI providers to honor the signal. The company described the step as moving toward an open web where people can publish and discover information and website owners have clearer, simpler ways to communicate their choices about how their content is used, and said it will continue to adapt its practices as those standards conversations evolve.
How the Signal Works
The signal is defined in an individual Internet-Draft titled Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00), authored by Kevin Kelley of Anthropic, dated October 30, 2026 and expiring May 3, 2027. The IETF’s datatracker notes that an individual Internet-Draft is not endorsed by the IETF and has no formal standing in the IETF standards process.
Under the draft, a service operator binds usage preferences, such as “no AI training,” to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line. The draft introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of the robots.txt standard published as RFC 9309 are unchanged. As a result, the draft says, existing parsers deployed across the internet will continue to interpret traffic-management signals as they did previously and will ignore policy-token groups as they would any group naming a user agent they do not recognize.
The draft defines the cross-vendor policy token __usage_ai-training. Its example group pairs the line “User-Agent: __usage_ai-training” with “Disallow: /” and “Allow: /press-kit/”, which expresses a preference against AI training for all paths except those under /press-kit/. Wildcard groups, meaning “User-Agent: *”, must not be interpreted as matching the policy token, and if no group in a file contains the token, then no preference has been expressed.
The draft also states that the policy token is not a directive for crawling behavior. Content-usage preferences and crawler traffic-management signals are independent and orthogonal, it says, and private agreements may override preferences expressed through the mechanism. Its stated goals include path-specific bindings, minimal distance from RFC 9309, no large-scale duplication of rules, unchanged behavior for parsers that do not implement the mechanism, and consistent interpretation across consumers.
The Vocabulary Behind the Signal
The token’s meaning tracks the working group’s vocabulary draft, A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-08), written by Paul Keller of Open Future and Martin Thomson of Mozilla and last updated in September 2026 as a working group document intended as a Proposed Standard. That vocabulary defines AI Training as using an asset to modify the learned parameters of a generative AI model, excluding the training of models used for exclusively non-generative tasks, and it defines a generative AI model as one used, or made available for use, to generate synthetic content in one or more modalities.
The vocabulary also defines a separate AI Use category, covering assets used as input to a generative AI model rather than provided directly by a user, and a Search category for applications whose primary purpose is selecting assets and directing users to their locations. Each category receives a preference value of allowed, disallowed, or unknown, and where statements conflict, the most restrictive preference applies. The vocabulary draft states that it does not ensure preferences are followed and leaves the choice of whether to follow them to the recipient, and a note to readers cautions that its contents do not yet reflect working group consensus. One section, whose inclusion is still tracked as an open issue, states that complying with training preferences includes conveying the preferences associated with training assets along with a model when it is distributed.
The training-signal draft’s acknowledgments name working group chairs Mark Nottingham of Cloudflare and Suresh Krishnan of Cisco, vocabulary editors Thomson and Keller, and responsible area director Mike Bishop of Akamai. They also list individual participants, with affiliations noted for identification only because IETF participants contribute as individuals, from organizations including OpenAI, Google, Microsoft, Meta, Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation, and Creative Commons.
OpenAI’s Existing Crawler Controls
OpenAI’s crawler documentation, which the company said will remain in effect alongside the new signal, describes GPTBot as the agent that crawls content that may be used in training its generative AI foundation models; disallowing GPTBot indicates a site’s content should not be used for that training. OAI-SearchBot surfaces websites in ChatGPT’s search features, and sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. OAI-AdsBot validates the safety of web pages submitted as ads on ChatGPT, and the data it collects is not used to train generative AI foundation models. ChatGPT-User handles page visits triggered by user actions in ChatGPT rather than automatic crawling, and the documentation notes robots.txt rules may not apply to those visits.
Each setting is independent of the others, according to the documentation, so a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot for training. Search adjustments can take about 24 hours after a site’s robots.txt update. OpenAI said it will share implementation details for the shared signal when it implements it in the coming months.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.