How to Block AI Training Without Blocking Google Search
Cloudflare AI crawler controls now let you treat search, training, and agents as three different decisions. That matters if you want Google to keep indexing your pages while you refuse to feed model training.
Until recently, mixed-use crawlers forced a bad tradeoff. One bot collected content for a search index and for training. Blocking the bot meant disappearing from search. Allowing it meant donating the archive. Cloudflare’s September 2026 update was built to end that bind.
This is a setup guide for site owners in the US, UK, Canada, Germany, Saudi Arabia, the UAE, South Africa, and Ethiopia. It is not legal advice, and crawler rules change. Check the live labels in your dashboard before you lock a policy.
What changed in 2026
On 15 September 2026, Cloudflare launched a setting named Disallow AI Training and an Accountable label for mixed-use crawlers. In Cloudflare’s own press release, Apple, Google, and Microsoft are treated as Accountable when they give site owners a way to opt out of training without opting out of search.
Google’s version of that split is Google-Extended: you can disallow training-related use while Googlebot still crawls for Search. Google has said that opting out of Google-Extended does not, by itself, change ordinary search ranking.
Cloudflare also replaced the old “Managed robots.txt” helper with Bot Preference Sync. You set Search / Training / Agent once. Cloudflare writes matching robots.txt preferences and can enforce many of them at the edge.
Cloudflare reported that mixed-use crawlers were 36.6% of verified crawler traffic on its network, that fewer than 1% of sites block search crawlers, and that 17% restrict AI training. Treat those as Cloudflare’s network snapshot, not a promise about your niche.
The three switches (and the one people get wrong)
| Control | What it is for | Safe default if you sell ads or publish original work |
|---|---|---|
| Search | Classic indexers such as Googlebot and Bingbot | Allow |
| Training | Crawlers that collect text to train or fine-tune models | Disallow AI Training (not “Block”) |
| Agents | Bots that browse to complete tasks for a user | Decide by business model |
The mistake is using Block on Training when you meant Disallow AI Training. On mixed-use crawlers, Block can stop Googlebot, Bingbot, or Applebot entirely. Disallow AI Training tells Accountable crawlers “search is fine, training is not,” and blocks training-only crawlers that do not offer that split.
If your pages carry ads, Cloudflare’s recommended starting point is: search on, training disallowed, agents blocked on ad-carrying pages. Other sites often leave all three allowed. Those are recommendations, not laws.
Step-by-step: Cloudflare dashboard
- Log in to Cloudflare and open the zone for your live hostname.
- Go to the AI / bot preferences area (the label may say AI Crawl Control, Bots, or similar).
- Set Search to Allow unless you truly want to leave Google and Bing.
- Set Training to Disallow AI Training.
- Turn Bot Preference Sync on so robots.txt stays in step with the dashboard.
- Save. Wait a few minutes, then open
https://yourdomain.com/robots.txt.
You should see a no-training preference for Google-Extended and other cooperating tokens, while Googlebot is not disallowed site-wide. If you already maintain a hand-written robots.txt, Bot Preference Sync prepends its rules. Do not paste a User-agent: Googlebot / Disallow: / rule “just to be safe.” That is how you leave Search.
If you are not on Cloudflare
You can still split search from training in robots.txt:
User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: Googlebot Allow: /
Confirm current token names in each vendor’s docs. Tokens get renamed. A stale name is a rule that nobody reads.
Search Console still needs a crawlable, indexable page with a snippet. Google’s AI features documentation is the source for AI Overviews and AI Mode eligibility. That is a different switch from training.
Search summaries are not the same as training
AI Overviews and AI Mode can quote or summarize a page even when training is disallowed. Google now offers a Search Console control for generative AI features. Using it can remove impressions from those features. It is not a ranking penalty for classic blue links, according to Google, but you will not get that AI-feature traffic.
If your goal is “keep ranking, refuse model training,” start with Cloudflare AI crawler controls plus Google-Extended. Only use the Search generative-AI opt-out if you also want to leave Overviews and AI Mode.
For the content side of AI Search, see how to optimize for Google AI Search.
Checks before you walk away
- Fetch robots.txt as Googlebot would see it, including any CDN overlay.
- In Search Console, confirm coverage is not falling after a week.
- Watch Cloudflare analytics for Googlebot vs training-only bots.
- If you use WordPress, do not let a “block all bots” security plugin override Cloudflare.
- Hosting cost and CDN choice still affect crawl budget. Our WordPress hosting comparison and cloud hosting cost guide cover the money side.
FAQ
Will Disallow AI Training hide me from Google?
It should not, if Search stays allowed and Googlebot is not disallowed. Training opt-out is a separate token.
Does this stop every AI company?
No. Robots.txt is honored by crawlers that choose to honor it. Cloudflare can block many known training bots at the edge. Unknown scrapers still need rate limits, WAF rules, and legal notices.
What about Bing?
Microsoft has been building site-level no-training support. Until a vendor documents that it honors the directive, do not assume Bing behaves like Google-Extended. Read Bing’s current webmaster help before you rely on it.
Is llms.txt required?
Google Search says it does not use llms.txt for ranking or AI features. Other tools might. It will not replace Cloudflare AI crawler controls.
Bottom line
Set Search to allow. Set Training to Disallow AI Training, not Block. Sync robots.txt. Then verify Googlebot still fetches the homepage. That is the whole point of the 2026 controls: stay findable, stop donating the corpus by accident.
