By the Smili Phone team
If you run a website, you used to need a different rule for every AI company: please do not train your models on my pages. On October 30, 2026, that changed. Google, Microsoft, OpenAI and Anthropic each announced they will honor one shared robots.txt rule for AI training preferences, so a single entry can now speak to all four.
The rule comes from a proposal filed the same day with the Internet Engineering Task Force. Anthropic engineer Kevin Kelley wrote it. It is still a proposal, not a finished standard, and it expires on May 3, 2027 unless it is revised. The commitments behind it are real, on different schedules.
What the rule actually says
Robots.txt is a small text file that sits at the root of a website and tells automated programs what they may do there. The new entry looks like this:
- User-Agent: __usage_ai-training
- Disallow: /
Those two lines ask that a whole site stay out of AI training. You can also limit the rule to part of a site by listing a single folder instead of the slash. The unusual part is the name on the first line. It is not a crawler, and no bot ever sends it. It is a shared label that stands for one use, training, so every participating provider reads the same preference the same way.
When each company starts honoring it
Microsoft gave the firmest date. Bing will start honoring the new preferences on February 1, 2027, as it crawls pages. Google says it aims to support the control in 2027, and its crawler documentation narrows that to early 2027. OpenAI has said its support will arrive in the coming months, without a published date. Anthropic, whose engineer wrote the proposal, has already started building support into its systems.
That staggered timing matters. Adding the rule today states a preference, not an instant switch. It takes effect as each company rolls out support and recrawls your pages, and it does not reach back to copies already collected.
What it does not do
This is where most confusion lives. The rule is a training preference, not a crawl block. It does not stop bots from visiting your pages, and it does not remove a site from Google or Bing search results. Search rankings are unaffected.
It also does not, by itself, stop a page from being quoted in a live AI answer. That is a separate use, and Microsoft handles it with a separate page level setting. Google has said that if a site opts out through the shared rule without setting Google-Extended explicitly, Google plans to treat it as a Google-Extended disallow.
Should a small business use it?
There is no single right answer. If your pages are original work you do not want feeding future models, the shared rule is now the simplest way to say so. If your business depends on being found and quoted in AI answers, opting out of training alone should not hurt that, but blocking live answers could.
What this means for everyone else
Even if you never touch a robots.txt file, this shift matters. It is the first time the biggest AI providers have agreed on one shared way for a site owner to express a training preference. Expect revisions as the rollout lands.
If you run a small site in Escondido and want help checking your robots.txt file, bring your device by the shop. We can walk through it with you.
Related: OpenAI's agent push at DevDay · Google's agent gets its own email
