Cloudflare AI Bot Settings: Block vs Disallow AI Training

👁 120 reads

Since September 15, 2026, Cloudflare divides AI bots into three groups - Search, Training and Agent - and lets you choose separate settings for each. The most important change: the Block option now also blocks Googlebot, Bingbot and Applebot, so if you only want to prohibit model training, you should choose the new Disallow AI Training option. Below we explain all the settings and suggest what to choose for blogs, stores and ad-supported sites. More about the company: Cloudflare - guide.

Why Cloudflare changed this

The biggest problem until now was “mixed” bots - a single robot that simultaneously indexes pages for search and collects content for training models. Googlebot is the best example. Block it to prevent Gemini from learning on your content, and you disappear from Google. That’s why for a long time “Block” in Cloudflare simply didn’t cover such bots.

According to Cloudflare, less than 1% of sites on its network block search engine bots, but 17% enable some form of training block. The robots.txt file alone won’t solve this - anyone can publish one, but it can’t verify who’s actually fetching the page or stop a bot that ignores it. Cloudflare sees the traffic, so it can both publish the preference and enforce it.

All settings in one table

Disallow AI Training blocks, among others, pure training robots from Amazon, Anthropic, Meta and OpenAI. Cloudflare notes that blocking them doesn’t affect search engines.

What happened to previous settings

If you didn’t change anything, Cloudflare migrated your old settings automatically:

If you previously set separate toggles and had Block or Block on pages with ads for Training, you now have Disallow AI Training. Watch out for one thing: if you manually set Block for Training or Search, you’ll block Googlebot. That’s the easiest way to lose Google traffic in a week.

Default settings for new domains

When adding a domain, Cloudflare asks whether you monetize with ads and suggests one of two sets:

The logic is simple: ads only earn money when a human sees the page. Training replaces the visit with an answer, and an agent fetches the page when no one is looking at ads.

Bot Preference Sync - robots.txt that writes itself

This feature, available since August 21 on all plans, adds a section at the beginning of your robots.txt that matches your dashboard settings. Existing rules remain, new ones go above them, between # BEGIN Cloudflare Bot Preference Sync and # END Cloudflare Bot Preference Sync markers. Cloudflare updates the bot list itself based on its BotBase database, so you don’t have to track which new robots appeared this quarter.

When to disable sync: if you have individual agreements with a specific AI company and want to allow it despite the general rule. Sync works on entire categories, it doesn’t read your custom firewall rules.

What to choose - our recommendations

Tutorial blog, no ads

• Search: Allow, Agent: Allow.
• Training: Allow or Disallow AI Training - it’s a matter of principle. You won’t lose traffic either way.
• Consider signing up for the Pay Per Use beta if you write specialized, fresh content - we described it in Pay Per Use from Cloudflare.

Ad-supported site (AdSense and similar)

• Search: Allow, Training: Disallow AI Training.
• Agent: worth considering. Block on pages with ads will stop agents like ChatGPT fetching the page on user request. You’ll gain less useless traffic, but lose some citations with links in chats. If you care about AI visibility (GEO), leave Allow.

Online store

• Everything on Allow. A shopping agent comparing prices is a potential customer.

Paywalled site

• Search: Allow, Training: Disallow AI Training, Agent: Block or Block on pages with ads - an agent bypassing the paywall is a real risk.

How to set this up step by step

1. Log in to the Cloudflare dashboard and select your domain.
2. Go to AI bot settings (AI Crawl Control section or bot settings in the security tab - Cloudflare changes menu names fairly often).
3. For Search, Training and Agent, choose options according to the table above.
4. Enable Bot Preference Sync if you don’t manage robots.txt manually.
5. Open yourdomain.com/robots.txt and check if the Cloudflare section appeared.
6. After a week, check AI bot statistics and Google Search Console to see if the number of indexed pages changed.

We didn’t have access to a specific domain’s dashboard while writing this, so we’re giving button names based on Cloudflare posts from August 21 and September 15, 2026. If yours looks different, look for three toggles: Search, Training and Agent.

What’s next: control over AI summaries

Cloudflare announced that by early 2027 it will let you set how much content can appear in AI summaries (for example, AI Overviews in Google) - with one setting in Cloudflare, instead of separately with each operator. “Accountable” status already requires Google, Apple and Microsoft to offer the ability to opt out of summaries directly with them.

General principles for blocking AI robots, regardless of Cloudflare, are described in how to block AI bots on your own site, and the history of changes from August - in Cloudflare blocked AI bots on my site.

Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →

Sources

• Cloudflare - Have it both ways: Disallow AI Training (September 15, 2026)
• Cloudflare - Bot Preference Sync (August 21, 2026)
• Cloudflare - The Internet has a second audience (September 30, 2026)

Kinetyra