Four of the world’s largest artificial intelligence companies have promised publishers a single, standard way to keep their content out of AI training data. OpenAI, Google, Microsoft and Anthropic pledged on Friday to support one shared machine-readable signal that websites can use to say no to training scraping, instead of the patchwork of separate blocking commands publishers must maintain today.
The announcement lands at a moment of open tension between publishers and AI developers. News outlets and other websites have complained for years about a surge in bots scraping their articles to feed AI models, and several publishers are already fighting the practice in court. Until now, a publisher that wanted to block training crawls had to write different rules for different companies’ bots — and still could not be sure every crawler would respect them.
Key Takeaways
- OpenAI, Google, Microsoft and Anthropic will honour one shared robots.txt signal for AI training opt-outs.
- The signal, called usage_ai-training, replaces the need for separate blocking rules for each company’s crawlers.
- Rollout is staggered: Microsoft starts on February 1, 2027, Google follows during 2027, and OpenAI says its support arrives in the coming months.
One signal instead of many rules
The shared token is called usage_ai-training. It lives in the robots.txt file — the standard file site operators already use to publish rules for automated crawlers — and it is defined in an Internet Engineering Task Force (IETF) individual draft titled “Machine-Readable Training Preference Signal”, authored by Kevin Kelley of Anthropic and dated October 30, 2026.
The idea is deliberately simple: a publisher enters one standardised instruction, and any AI company that adopts the protocol recognises and honours it. That replaces a system in which publishers had to implement multiple blocking commands for every bot that crawls the web to collect data. The draft is an early stage of IETF standardisation — documented and open for review, but not yet adopted as a formal standard — and the rollout picture comes from press reports and company blogs rather than a signed standard.
A publisher enters one standardised instruction, and any AI company that adopts the protocol recognises and honours it.
What the new control will — and will not — do
Publishers should read the fine print carefully, because the new signal has clear limits. According to Microsoft, the setting will apply only to future training of models and does not prevent bots from crawling a site otherwise. Content scraping will also be allowed by default until a publisher changes the setting — the opt-out must be switched on, not off.
OpenAI adds that the new signal will not replace the controls it already provides, so existing robots.txt rules stay in place, and the token does not guarantee exclusion from training. In other words, the pledge is a voluntary commitment by the companies involved, not a legal ban on scraping, and its strength depends on how widely it is adopted and honoured.
- One shared token, usage_ai-training, for all four companies’ crawlers
- Applies to future AI training only — it does not stop ordinary web crawling
- Scraping stays allowed by default until a publisher switches the setting on
- Existing robots.txt controls remain in place alongside the new signal
Staggered rollout through 2027
The four companies are not moving on the same timetable. Microsoft will implement the signal beginning February 1, 2027, and Google sometime in 2027. OpenAI said it will roll support out in the coming months and publish a further update with implementation details when it does. Reports say Anthropic has already begun honouring the token, though those timelines were not all confirmed in the companies’ own primary materials at the time of writing.
The announcement also sits inside a wider week of scrutiny for the AI industry. Investors have been questioning the massive spending behind AI development, and tech shares slid this week as AI spending fears gripped the market. At the same time, trust in AI tooling has been shaken by security incidents, including the tampered Claude Code extension that reached thousands of Cursor users. Against that backdrop, a publisher-friendly gesture carries real public-relations value for the AI giants — but publishers will judge it on enforcement, not announcements.
Why publishers wanted this
For publishers, the stakes are straightforward. Their reporting costs money to produce, and AI models trained on scraped articles can then answer readers’ questions without ever sending them to the original site. A single, respected opt-out gives smaller publishers in particular a realistic way to protect their work, since few newsrooms have the technical staff to track every new crawler and write a rule for each one.
The move also matters for the broader web. If the shared signal gains traction, it could become the default vocabulary for training preferences — the same way robots.txt itself became the default vocabulary for crawling decades ago. OpenAI has encouraged other AI providers to honour the signal as well, and framed the step as strengthening its participation in IETF standards discussions. The more providers that adopt it, the more useful one line in robots.txt becomes, which is exactly the network effect publishers have been asking for.
Still, sceptics will note what the announcement does not cover. It does not settle the legal questions already before the courts, it does not compensate publishers for content already scraped and used in training, and it relies on voluntary compliance. A publisher that opts out is trusting each AI company to keep its word — a trust that, given the industry’s week, is in short supply. The related question of how AI companies fund their enormous ambitions remains open too, as OpenAI’s latest funding round reportedly came in below its original target amid the tech sell-off.
What to Watch Next
Watch for the first implementation details from OpenAI in the coming months, and for Microsoft’s February 1, 2027 start date, which will be the first hard test of whether the shared signal works in practice. Watch whether more AI providers join the pledge, whether the IETF draft advances toward a formal standard, and whether publishers report that their opt-outs are actually being respected. The announcement promises publishers a simpler switch; the months ahead will show whether that switch does what it says.