AI Giants Back Standardized Scraping Opt-Out
AI-generated image · Bay Street Wire
OpenAI, Google, Microsoft, and Anthropic agree to a shared 'policy token' to streamline how publishers block AI training.
Major AI developers are moving toward a unified system for publishers to opt out of data scraping, shifting away from a fragmented process where site owners must set preferences for each company individually.
As reported by The Verge, OpenAI, Google, Microsoft, and Anthropic have pledged to adopt a new standard that allows publishers to apply AI training preferences across multiple crawlers simultaneously. The proposed "policy token" signal, introduced by Anthropic technical lead for standards Kevin Kelley as part of the Internet Engineering Task Force (IETF), will be integrated into a website's robots.txt file.
Anthropic has already adopted the signal, while Google and Microsoft plan to provide support next year; OpenAI expects to implement the change in the coming months. According to The Verge, the companies have stated they will restrict their AI crawlers from accessing sites that display the shared token.
Microsoft is introducing more specific options, allowing publishers to choose between opting out of AI training or AI grounding—the practice of using verified sources to minimize hallucinations. Microsoft clarified that these preferences will not affect a site's visibility in Bing search results.
This move comes as publishers seek ways to manage the surge of AI crawlers. The Verge notes that other efforts to curb unauthorized scraping include Cloudflare blocking AI crawlers by default and the Really Simple Licensing (RSL) standard, which enables publishers to charge visiting crawlers.