September 15, 2026: Cloudflare's crawler split and how to configure Search, Agent and Training

Summary. On July 1, 2026 Cloudflare gave every customer, including the $0 Free tier, three separate switches for AI traffic: Search, Agent, and Training. On September 15, 2026 Cloudflare changed what it had announced. New domains are now offered presets: a site that earns from ads gets Search on Allow, Training on a new setting called Disallow AI Training, and Agent on Block on pages with ads, while other sites get Allow on all three. Disallow AI Training publishes a no-training preference in robots.txt, blocks other training crawlers, and keeps Googlebot, Applebot, and BingBot crawling for search. The catch has moved: Block and Block on pages with ads now stop those three search bots as well. Bots already passed humans on the network at 57.5% to 42.5% by mid-2026, and Cloudflare Radar for 25 to 31 October 2026 put Anthropic at about 517 pages crawled per referral and OpenAI at about 1,900, against Google's 4.8. This guide shows publishers, SEO and GEO leads exactly which setting to choose for each of the three switches, how to keep Google Search access, and what to check now that the change is live.

Update, 2 November 2026. Cloudflare changed its plan on September 15, 2026, so this guide has been corrected. A new Training setting, Disallow AI Training, keeps Googlebot, Applebot, and BingBot crawling for search, and the opt-out described in July is no longer needed. Block and Block on pages with ads now stop those crawlers. New domains are offered presets rather than a single default, and Cloudflare says earlier Training blocks move to Disallow AI Training. Our AI crawler monetisation comparison covers what to do next.

Cloudflare sits in front of more than 20% of web domains, so a default change on its network is a change most site owners inherit whether they read the release notes or not. The company framed the update as its second "Content Independence Day," one year after it first let site owners block AI training crawlers. Matthew Prince, co-founder and CEO of Cloudflare, tied the move to raw traffic numbers: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge." The practical result is that a setting you never touched now has three positions instead of one, and the wrong position can pull you out of Google.

What changed on July 1, 2026

Cloudflare replaced the single "Block AI bots" toggle with a taxonomy built on what a bot does rather than whether it counts as "AI." Its reasoning is that "AI can be in anything" now that Google Search answers questions directly on the results page, so a label like "AI bot" ages badly. The three classes it wants every customer to manage are Search, Agent, and Training. All three are configurable today on every plan, from the Free tier at $0 to Business at $250 per month billed monthly, and Enterprise.

The definitions matter, because your setting keys off them:

Cloudflare also launched supporting pieces the same day: BotBase, a searchable directory of tracked bots for Enterprise Bot Management; a Monetization Gateway that lets sites charge bots for access and settle in stablecoins over the x402 protocol; and a content-use signal covering what a bot may keep after it crawls. Those are context, and the Monetization Gateway has since entered closed beta for eligible US-based sellers and buyers. The setting that changes traffic for most sites is the Training control.

What changed on September 15, in plain terms

Two things changed on September 15, 2026, and they replaced the plan Cloudflare set out in July.

First, new domains onboarding to Cloudflare are offered one of two presets, with Bot Preference Sync on in both. A site that earns from ads gets Search on Allow, Training on Disallow AI Training, and Agent on Block on pages with ads. Any other site gets Allow on all three. You can change these during onboarding or at any time after it. Cloudflare's logic is that ad revenue depends on a person seeing the page, which training and agent fetches do not provide.

Second, and this is the part that trips people up, the Training control now has four positions that treat multi-purpose crawlers differently. Disallow AI Training publishes a no-training preference, blocks every other training crawler, and leaves Googlebot, Applebot, and BingBot crawling for search. Block and Block on pages with ads now apply to those three as well, so either one affects search. There is no ads-only version of Disallow AI Training, because Cloudflare says the list of ad pages is too large to write into robots.txt.

You no longer need the opt-out Cloudflare described in July. Cloudflare says existing settings carry over on their own: the legacy Block AI Bots setting, and earlier Training selections of Block or Block on pages with ads, move to Disallow AI Training. If you want mixed-use crawlers gone entirely, you now have to select Block. That single choice is the difference between keeping Googlebot and losing it, and it is the first thing most ad-supported publishers should check. Cloudflare's Block AI Bots docs page, last updated July 1, 2026, still describes the earlier plan, so follow the September 15 post.

One gap remains. Until Microsoft's robots.txt support arrives, Disallow AI Training does not pass a no-training preference to Bing, so Cloudflare points site owners to Bing's NOARCHIVE tag. Microsoft says it will honour its new shared training preference in robots.txt from February 1, 2027, and will stop reading NoArchive as a training preference after March 1, 2027.

The Googlebot trap

Google does not run a separate training-only crawler that you can block in isolation. Its flagship Googlebot crawls for Search, and that same index feeds AI Overviews and AI Mode. Google offers Google-Extended as an opt-out for training its Gemini apps and Vertex AI, and using it does not remove you from Google Search. Cloudflare classifies Googlebot itself as multi-purpose, so your Training setting decides what happens to it. Choose Disallow AI Training and Googlebot keeps crawling for search. Choose Block or Block on pages with ads at the Cloudflare layer, and you block the crawler that keeps you in Google.

This is why the same setting helps one site and hurts another. A newsroom that wants payment for training data may refuse training, while an ecommerce store may not mind it. Neither should choose Block unless it wants to leave Google and Bing. The table below shows how the major crawlers land under the new settings, based on Cloudflare's September 15 post.

The design goal is transparency: Cloudflare wants operators to run separate crawlers for separate jobs, and it argues that the ones that already do, such as OpenAI and Anthropic, are easier for site owners to manage than a single blended bot. Whether you agree with the pressure campaign or not, the operational fact stands. If your revenue depends on Google referrals, choose Disallow AI Training rather than Block, or you risk your own indexing.

Why Cloudflare is forcing the question: the crawl economics

The reason a CDN is rewriting crawler defaults at all is that the old bargain broke. For thirty years the deal was simple: crawlers took your pages and sent back readers. AI crawling inverted that. Cloudflare Radar data through October 2026 quantifies the gap, and the numbers are the strongest argument in the whole debate.

Read the dates carefully. OpenAI ran at about 251 to 1 in July 2026 and about 1,900 to 1 for 25 to 31 October 2026, while Anthropic moved from about 1,917 to 1 to about 517 to 1 over the same months. One week is noisy, and Cloudflare says native apps can make these ratios overstate the gap. Even so, both labs crawl far more per referral than Google, and that week ClaudeBot was 81.4% of Anthropic's crawling and GPTBot 68.5% of OpenAI's. Training returns almost nothing, and most AI crawling is training rather than search. That is the case Cloudflare is making with its defaults: keep the class that sends readers, gate the classes that do not.

For context on how little of the click now reaches publishers even when a page does rank, our analysis of ranking versus AI Overview citation data and the keyword selection framework for click survival both show why raw impressions stopped translating into visits in 2026.

How to configure the three classes, step by step

You control all of this from the Cloudflare dashboard, and you can back it with a robots.txt signal. Here is the deliberate setup.

1. Open your zone in the Cloudflare dashboard and go to Security, then Settings. Filter by Bot traffic and select Configure AI bot policies. The AI traffic controls for Search, Agent, and Training are there for every plan, including Free.

1. Decide per class. Allow Search unless you specifically want to charge for indexing. Choose Agent based on whether you want real-time assistants fetching your pages. Choose Training based on whether you are willing to have your content absorbed into models without payment. Training has four positions: Allow, Disallow AI Training, Block on pages with ads, and Block.

1. If you run ads and depend on Google, set Training to Disallow AI Training, not Block or Block on pages with ads, so Googlebot, Applebot, and BingBot stay allowed for search. Cloudflare exposes this directly in Security settings.

1. Back the choice in robots.txt with Bot Preference Sync, which replaced Managed Robots.txt on September 15. It writes preferences, including Content Signals, that AI operators are meant to honour, and adds nothing while every policy is on Allow. The new content-use field sets how far a bot may go: immediate (use nothing), reference (index, excerpt, link back, the default), or full (summarize and reproduce).

With Training on Disallow AI Training, the managed block starts like this, then lists the crawlers it disallows:

            User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
          

A signal is a stated preference, not a hard block; the dashboard setting is what actually allows or denies traffic at the edge. Use both so your intent is legible to operators that respect robots.txt and enforced against those that do not. If you already run bot rules, sanity-check them against your broader bot-detection setup; our note on Cloudflare's session-based bot detection covers how these layers interact.

Which setting each kind of site should pick

There is no universal answer, because the three classes trade discoverability against control. Map your choice to what the site is for.

Choosing Disallow AI Training over Block is the single most important control for anyone whose traffic comes from Google. Set it deliberately rather than inheriting a default that was designed for a different kind of site. For the wider strategy of staying visible as Google shifts users into AI Mode by default, and for how structured, citable content earns AI Overview citations, crawler access is the precondition: an engine cannot cite a page it was blocked from reading.

India-specific considerations

Indian publishers and D2C brands sit in the same trap with an added compliance layer. Ad-monetized Indian news and content sites that rely on Google referral traffic should confirm Training is on Disallow AI Training rather than Block, or risk losing organic reach at the same time the market is shifting to AI answers. For Indian SaaS and fintech firms, the Agent class is the one to watch, because real-time assistants acting on a user's behalf increasingly complete tasks inside web apps, and blocking them can break legitimate automated flows.

On the data side, the content-use levels connect to India's Digital Personal Data Protection Act. Any page that exposes personal data should lean toward immediate or reference rather than full, so a model does not reproduce personal information verbatim. Teams building consent and data-handling flows can pair this with our DPDP engineering playbook for Indian startups. The crawler setting and the privacy obligation point the same way: control what leaves your site and how it can be reused.

Your checklist now that the change is live

Do these in order. First, confirm whether your key pages display ads, because Block on pages with ads and the ad-funded preset only matter if they do. Second, if Google traffic matters, confirm in Security settings that Training reads Disallow AI Training, not Block. Third, set Search, Agent, and Training deliberately per the site-type table rather than leaving them where the migration put them. Fourth, turn on Bot Preference Sync and check your robots.txt for a use=reference signal so your intent is stated as well as enforced. Fifth, if you run premium content, evaluate paid access such as the Monetization Gateway, in closed beta for eligible US-based sellers and buyers, instead of a flat block. Finally, watch your server logs and Cloudflare analytics for two weeks after any change to catch any crawler you blocked by accident.

FAQ

How eCorpIT can help

eCorpIT is a senior-led engineering and digital marketing organisation in Gurugram that helps publishers and product teams configure crawler policy without losing search or AI-search visibility. We audit your Cloudflare bot settings, set the Search, Agent, and Training classes against your business model, and align robots.txt Content Signals with your DPDP obligations. If the September 15, 2026 change affects your traffic, contact our team or read more about our GEO and AEO optimisation service.

References

1. Cloudflare Blog — Your site, your rules: new AI traffic options for all customers (July 1, 2026)

1. Cloudflare Blog — Have it both ways: stay discoverable in search while disallowing AI training (September 15, 2026)

1. Cloudflare Press — Cloudflare allows the agentic Internet to flourish: your content, your rules

1. Cloudflare Blog — Content Independence Day, one year on: building the business model for the agentic Internet

1. Cloudflare Blog — Making AI search smarter

1. Cloudflare Blog — Announcing the Monetization Gateway

1. Cloudflare Blog — Monetization Gateway beta: charge AI agents for consumption with HTTP 402 (September 30, 2026)

1. Cloudflare Blog — Google's AI advantage: why crawler separation is the only path to a fair Internet

1. Cloudflare Developer Docs — Block AI bots

1. Cloudflare Developer Docs — robots.txt setting, Bot Preference Sync (last updated October 22, 2026)

1. Cloudflare Developer Docs — BotBase

1. Microsoft Bing Blog — Your content, your choice: clearer preferences for training and grounding (October 30, 2026)

1. TechCrunch — Cloudflare's new policy pushes AI companies to pay for publishers' content

1. Search Engine Journal — Cloudflare's AI crawler rules can block Googlebot

1. Content Signals — Content Signals specification

1. Cloudflare Radar — AI Insights, crawl-to-refer ratio, last 7 days, 25 to 31 October 2026 (read November 1, 2026)

1. Cloudflare Blog — The crawl before the fall of referrals: understanding AI's impact on content providers

1. TechnologyChecker — ChatGPT statistics, full-month Cloudflare Radar pulls for July 2026

1. SEOmator — GEO data report 2026: crawl-to-refer ratios for AI crawlers and LLM bots

1. Spendbase — Cloudflare pricing explained: Free, Pro, Business and Enterprise plans

Last updated: November 2, 2026.