AI 模型和平台
Google计划支持共享机器人.txt AI培训控制
在 Google 上的首选来源中添加 Unite.AI
2026年10月30日，Google 宣布计划支持在机器人.txt中共享AI训练控制，给网站所有者一个单一的方式，在多个AI供应商中表达培训偏好。 承诺与Microsoft和OpenAI的同一天通知同时作出。
Google 将如何应用控制
计划由约翰·穆勒(John Mueller),Google搜索律师在Google搜索中央博客文章中阐述。 该控制基于互联网工程任务组AI优惠工作组(IETF)的工作，并基于Google-Extend，即它自2023年以来提供的AI培训控制Google 称。
网站所有者不会为每个AI开发者管理单独的指令，而是能够使用机器人内部共享的政策符.txt来设定参与的AI提供者的广泛偏好。 当一个网站使用符号选择退出培训，而未明确指定Google-Extend后，Google计划对此进行解释，仿佛出版商否决了Google-Extend后。 其他AI提供者可以同样处理信号，因此所有者可以一并选择退出所有参与提供者或选择。
根据该帖子，现有的公司特定控制将继续与联合信号一起运行，因此网站所有者可以通过行业信号表达广泛的选择，同时仍然通过Google-Extend等控制对单个开发者进行定制指令。 特别允许Google-Extended的网站有资格在响应中出现，通过Gemini Apps的地平面接收点击。
Google 称提案反映了出版商、民间社会团体、监管者和其他利益攸关方的反馈，它们呼吁全行业对AI培训进行控制，而AI主要供应商的支持将产生一个在全行业发挥作用的持久标准。 该公司还提出称，与欧盟AI法案下的通用AI模型提供商的"欧盟业务守则"一致，该守则承认机器人.txt是网络出版商表达机器可读偏好的主要方法。 Google旨在与其他AI供应商一起支持新控制，在2027年和称，它将提供其最新进展。
OpenAI在同一天作出了平行承诺。 兹克 1Z的公告称公司将尊重常见的机器人.txt信号，旨在在未来几个月内实施，并将保持其现有的爬行控制并尊重。 OpenAI 称正在与AI优惠工作组的其他参与者一起宣布，并鼓励AI的其他供应商尊重信号。
信号后面的 IETF 草稿
该机制在由Anthropic的Kevin Kelley编写、日期为2026年10月30日的题为“可读培训优先信号”的互联网草案中作了具体规定。 该文件旨在用于标准轨迹，并将于2027年5月3日到期；根据IETF数据跟踪仪，该文件是一份未经IETF认可的单独呈文，在标准程序中没有正式地位。
在草案下，一个服务运营商通过写出普通机器人来将用法偏好绑定在路径上。 txt 组合按键为每个目的的政策符，usage ai-training，位于用户-Agent行。 除了/press-kit/下的路径之外，推出机制没有新的指令名称，也没有新的语法：路径语言，最长的相匹配优先级，以及机器人.txt标准的组结构，RFC 9309，没有变化，因此已经部署在互联网上的解析器会忽略他们不承认的政策-token组。 通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通。 txt文件没有 usage ai-training group 表示任何偏好。
该标志的含义取自AIPREF词汇对AI培训的定义："利用资产来修改遗传AI模型的所学参数". 该定义排除了专门用于非再生任务的模型的培训，即使该模型能够进行再生任务。 草案还强调指出，该信使表示倾向于使用内容，而不是对爬行行为的指令，尽管消费者可能利用偏好为爬行提供信息；具体表达方式应优先于一般表达方式，私人协议可能优先于表达的偏好。
Kelley 宣布 2026年10月30日IETF的AI控制邮件列表的草稿，写道Anthropic已经开始对其实施支持。 他说，该工作是对Anthropic现有爬行器控制的一种补充，而不是替代。
Microsoft的收养和过渡日期
Microsoft 宣布于同日被通过，Microsoft 称是第一家为AI培训和地取地提供发布器控制的公司，指2023年NoArchive和NoCache标记它推出。
Microsoft正采用对站点和路径的共享AI培训偏好，并通过meta标记和HTTP响应头来将它扩展至页面级别，同时增加了单独的页面级别AI定位偏好。 除了允许和Disallow之外，基于新增的首选是一个新的DisallowExplated选项，它限制对Microsoft操作体验的基于使用，例如AI Mode,AI Overview,Copilot，和Microsoft 365，同时排除了向第三方提供的地上服务Microsoft，例如Web IQ，并通过在Foundry的API访问来进行地上操作。 两种偏好都默认在不发表偏好时允许。 Microsoft警告说，选择Disallow或DisallowExtend用于打地基会降低AI体验中的品牌能见度；偏好不会改变传统Bing搜索结果的资格，也不会建立许可或支付安排。
新首选将取代"NoArchive"和"NoCache"进行训练和地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取地取于2026年12月15日在Bing Webmaster Tools开放临时地基过渡偏好，用于自当日起使用更老的标记的网站。 这些网站必须在2027年3月1日之前公布其新的首选，并记录暂时的首选，之后NoArchive和NoCache将不再被解读为培训或入选的首选。 临时优惠仍然有效，直到Microsoft重新绘制一页并发现新的地基优惠，或者直到2027年9月30日，以先到者为准。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
AI Models & Platforms
Google Plans Support for Shared Robots.txt AI Training Control
Add Unite.AI to your preferred sources on Google
Google announced on October 30, 2026 that it plans to support a shared AI training control in robots.txt, giving website owners a single way to express training preferences across multiple AI providers. The commitment arrived alongside same-day announcements from Microsoft and OpenAI.
How Google Will Apply the Control
The plan was laid out in a Google Search Central Blog post by John Mueller, a Google Search Advocate. The control is based on work from the AI Preferences working group at the Internet Engineering Task Force (IETF) and builds on Google-Extended, the AI training control Google says it has offered since 2023.
Instead of managing separate instructions for every AI developer, website owners will be able to use a shared policy token within robots.txt to set broad preferences across participating AI providers. When a website uses the token to opt out of training and has not explicitly specified Google-Extended, Google plans to interpret that as if the publisher had disallowed Google-Extended. Other AI providers can handle the signal similarly, so owners can opt out across all participating providers at once or pick and choose.
According to the post, existing company-specific controls will continue to function alongside the joint signal, so website owners can express a broad choice through the industry signal while still tailoring instructions to individual developers through controls such as Google-Extended. Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps.
Google said the proposal reflects feedback from publishers, civil society groups, regulators, and other stakeholders that have called for industry-wide controls on AI training, and that support across major AI providers would produce a durable standard that works across the industry. The company also said the proposal is consistent with the EU Code of Practice for general-purpose AI model providers under the EU AI Act, which recognizes robots.txt as a primary method for web publishers to express machine-readable preferences. Google aims to support the new control, along with other AI providers, in 2027 and said it will provide updates on its progress.
OpenAI made a parallel commitment the same day. OpenAI’s announcement said the company will honor the common robots.txt signal, aims to implement it in the coming months, and will keep its existing crawler controls in place and respected. OpenAI said it was joining other AI Preferences working group participants in the announcement and encouraged other AI providers to honor the signal.
The IETF Draft Behind the Signal
The mechanism is specified in an Internet-Draft titled “Machine-Readable Training Preference Signal,” authored by Kevin Kelley of Anthropic and dated October 30, 2026. The document is intended for the standards track and expires May 3, 2027; per the IETF datatracker, it is an individual submission that is not endorsed by the IETF and carries no formal standing in the standards process.
Under the draft, a service operator binds usage preferences to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token, __usage_ai-training, in the User-Agent line. The draft’s example group expresses a preference against AI training for an entire site except for paths under /press-kit/. The mechanism introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of the robots.txt standard, RFC 9309, are unchanged, so parsers already deployed across the internet will ignore policy-token groups they do not recognize. Wildcard groups, those with User-Agent: *, must not be interpreted as matching the policy token, and a robots.txt file with no __usage_ai-training group expresses no preference at all.
The token takes its meaning from the AIPREF vocabulary’s definition of AI training: “Using an asset to modify the learned parameters of a generative AI model.” The definition excludes the training of models used exclusively for non-generative tasks, even when the model is capable of generative tasks. The draft also stresses that the token states a preference about content use rather than a directive for crawling behavior, though consumers may use preferences to inform crawling; specific expressions should take precedence over general ones, and private agreements may override expressed preferences.
Kelley announced the draft on the IETF’s ai-control mailing list on October 30, 2026, writing that Anthropic has already started implementing support for it. He described the work as a complement to Anthropic’s existing crawler controls rather than a replacement.
Microsoft’s Adoption and Transition Dates
Microsoft announced its adoption the same day in a Bing blog post by Krishna Madhavan, Principal Product Manager for Microsoft Bing. Microsoft said it was the first company to provide publisher controls for AI training and grounding, pointing to the NoArchive and NoCache tags it introduced in 2023.
Microsoft is adopting the shared AI Training preference for sites and paths and extending it to the page level through meta tags and HTTP response headers, while adding a separate page-level AI Grounding preference. Alongside Allow and Disallow, the grounding preference adds a new DisallowExtended option that limits grounding use to Microsoft-operated experiences such as AI Mode, AI Overview, Copilot, and Microsoft 365 while excluding grounding services Microsoft provides to third parties, such as Web IQ and grounding through API access in Foundry. Both preferences default to Allow when no preference is published. Microsoft cautioned that selecting Disallow or DisallowExtended for grounding reduces brand visibility in AI experiences; the preferences do not change eligibility for traditional Bing search results and do not establish licensing or payment arrangements.
The new preferences will replace NoArchive and NoCache for training and grounding and take effect starting February 1, 2027, as Microsoft crawls. Temporary grounding transition preferences open in Bing Webmaster Tools on December 15, 2026 for sites using the older tags as of that date. Those sites must publish their new preferences and record a temporary grounding preference before March 1, 2027, after which NoArchive and NoCache will no longer be read as preferences for training or grounding. Temporary preferences remain in effect until Microsoft recrawls a page and discovers the new grounding preference or until September 30, 2027, whichever comes first.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.