On this page · 10 sections
Summary. Cloudflare published "Say it once: introducing Bot Preference SynC" at 23:19 UTC on 21 August 2026. The feature generates or rewrites your robots.txt from the Search, Agent and Training settings you already hold in the zone dashboard, and Cloudflare's docs now list it on all plans. Three details in that post matter more than the headline. It works only on category-wide policy, so it will not read your per-crawler custom rules. It is on by default for all new customers. When we first published, the documentation page it needed, at /ai-crawl-control/features/managed-robots-txt/, returned a 404, while the page that did exist - the legacy robots.txt setting doc - described a different mechanism that hard-wrote Disallow: / for 8 named crawlers. That setting page has since been rewritten for Bot Preference Sync and was last updated on 22 October 2026. If you run a site behind Cloudflare with a hand-maintained robots.txt, check whether your file still says what you think it says after Cloudflare's 15 September 2026 changes.
Update, 2 November 2026. Cloudflare changed the rule this page warned about. Since 15 September 2026 its Disallow AI Training setting publishes a no-training preference, blocks other training crawlers and keeps Googlebot, Applebot and Bingbot crawling for search. Block and Block on pages with ads now stop those three. Bot Preference Sync replaced Managed Robots.txt, and Cloudflare documented it on its robots.txt setting page, last updated 22 October 2026. Our AIPREF robots.txt guide and AI crawler monetisation comparison cover what to do now.
What Cloudflare actually announced
The mechanism is narrow and worth stating exactly. The sync itself only publishes text, although the Disallow policy for Training also blocks other training crawlers at the edge. It takes the AI bot policy you have already configured at zone level - Allow, Block on pages that serve ads, or Block everywhere for Search and Agent, plus Disallow for Training - and reflects that policy as text inside robots.txt.
If your origin already serves a robots.txt, Cloudflare prepends its block ahead of your content rather than replacing it, wrapped in two comment markers:
# BEGIN Cloudflare Bot Preference Sync
User-agent: TrainingBot1
User-agent: TrainingBot2
User-agent: TrainingBot3
User-agent: MixedUseBot-Extended
Disallow: /
...
# END Cloudflare Bot Preference Sync
That example is Cloudflare's own, shortened and anonymised in the post. The real user-agent list is pulled from BotBase, Cloudflare's bot database launched on 1 July 2026, and Cloudflare says it will "periodically update the list of bots that is added to robots.txt when you choose to Block or Disallow a given category". You do not pin that list. Cloudflare does, and it changes underneath you.
The stated goal is to close the gap between what a site declares and what it enforces. As the announcement puts it: "When your stated preferences and your enforced rules disagree, some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules." That is a fair description of the problem. It is also the reason the custom-rules carve-out below matters.
The part the announcement does not lead with
Three limits sit further down the post, and each one changes an operational decision.
Category-wide only. Cloudflare states plainly that Bot Preference Sync "is designed to tackle policy decisions made category-wide rather than case-by-case, it will not directly read from individual custom rules with more complex logic." If you have negotiated an exception for one named crawler - a licensing deal, a partner index, an internal scraper - that exception lives in a custom rule and will not appear in the generated file. Your robots.txt will then advertise a stricter policy than your edge actually enforces, which is the exact mismatch the feature was built to remove. Cloudflare's answer is to turn the sync off and hand-maintain the file.
On by default for new customers. The post says Bot Preference Sync "will be on by default" for all new customers. Read alongside the next paragraph, that means the sync is on, not that blocks are on: for non-publisher new customers Cloudflare "will not have any blocks or disallows added by default". The two statements are consistent, but they read as contradictory on a first pass, and the practical effect is that a newly onboarded zone has the sync on, but Cloudflare leaves your robots.txt file unchanged until at least one policy is set to Block, Block on pages with ads or, for Training, Disallow.
A new publisher default. At onboarding, a customer can select "I monetize from pages with ads on this domain", which sets Training to Disallow as the default. Since 15 September 2026 that preset also sets Agent to Block on pages with ads. That is an opt-in checkbox with a policy consequence, changeable at any time.
The docs caught up, in a different section
The AI Crawl Control documentation set still does not contain a Bot Preference Sync page. Requesting /ai-crawl-control/features/managed-robots-txt/ returns Cloudflare's 404 template. The navigation lists Analyze AI traffic, Manage AI crawlers, and Directives - the tab formerly called Robots.txt - and nothing else in that area. The Directives page itself carries a "Last updated Apr 23, 2026" stamp and still points its managed-status card at the bots/additional-configurations/managed-robots-txt/ page.
That page is where Bot Preference Sync is now documented, in a version last updated on 22 October 2026. When we first published it described the legacy setting. Its example file still shows a content-signal line and a blanket disallow for a named list:
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
The list in that example is 8 crawlers: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, and meta-externalagent. The legacy toggle has gone. To set this now, go to Security Settings, filter by Bot traffic, select Configure AI bot policies, set Training to Disallow, turn on Enable Bot Preference Sync, and save. Cloudflare says customers who had the legacy setting on migrate to the new system.
| Behaviour | Legacy managed robots.txt (deprecated 15 Sep 2026) | Bot Preference Sync (announced 21 Aug 2026) |
|---|---|---|
| Source of directives | Fixed list of 8 named crawlers | BotBase categories, periodically refreshed |
| Policy model | Block training, one setting | Search, Agent, Training set separately |
| Reads your custom rules | No | No - category-wide only |
| Placement in your file | Prepended above existing content | Prepended above existing content |
| Documentation | Replaced on the same page | Live page, all plans, updated 22 Oct 2026 |
What changed on 15 September
Bot Preference Sync publishes a preference. It does not decide who gets blocked at the edge. Your AI bot policies do, and what they mean changed on 15 September 2026.
Cloudflare's 1 July 2026 post said that blocking Training would also block multi-purpose crawlers, and that new domains would block Training and Agent on ad pages by default. On 15 September 2026 it replaced that plan. A new Training setting, Disallow AI Training, publishes the no-training preference, keeps Googlebot, Applebot and Bingbot crawling for search, and blocks every other training crawler. Block and Block on pages with ads now apply to those three as well, so either setting affects search. New domains are offered presets: ad-funded sites get Disallow AI Training for Training and Block on pages with ads for Agent, and other sites get Allow on all three.
If you turned on Block AI Bots in 2025 and have not looked since, Cloudflare says that setting moves to Allow for Search, Disallow AI Training for Training, and Block on pages with ads for Agent. Earlier Training selections of Block or Block on pages with ads move to Disallow AI Training too. There is no opt-out left to find. If you want those three crawlers gone entirely, you now have to select Block.
The 21 August post set out the escape hatch on the preference side. The Training Disallow option writes a "no training" preference for mixed-use crawlers rather than a hard block, and Cloudflare says cooperating mixed-use crawlers that meet four transparency requirements keep search access. Those four requirements, verbatim from the post, are that the bot respects a "no training" preference in robots.txt by any mechanism; gives site owners a way to opt out of AI summaries; provides URL-level visibility into which pages were made available for training plus metrics on search results; and can show publicly that disallowing training does not hurt traditional search results. Operators meeting them are tracked in the AI bot transparency section on Cloudflare Radar. Operators that do not are still blocked when you disallow training. The 15 September 2026 post calls this status Accountable and says Apple, Google and Microsoft meet it, each with time-bound commitments for what is still in development. It also counts the training crawlers of Amazon, Anthropic, Meta and OpenAI as Accountable, and blocks them under Disallow because they are separate from search. One gap remains: until Microsoft's robots.txt support arrives, targeted for early 2027, Disallow does not pass a no-training preference to Bing, so Cloudflare points site owners to Bing's NOARCHIVE tag.
So the preference layer and the enforcement layer still work differently: mixed-use crawlers get a robots.txt preference, and other training crawlers get an edge block. That is the whole problem for anyone measuring AI search visibility.
How to tell whether this is you
Four checks, none of which take longer than a few minutes.
- Fetch your own
robots.txtand look for# BEGIN Cloudflare Managed contentor# BEGIN Cloudflare Bot Preference Sync. Either marker means Cloudflare is writing part of your file.
- Look for a
Content-signal:line. Cloudflare's managed content setssearch=yes, ai-train=no, use=reference. Theusefield is an optional extension Cloudflare describes as under test, with three values:use=immediate(interact, store and reuse nothing),use=reference(index, excerpt, and link back), anduse=full(summarize and reproduce).
- Open Security Settings, select Configure AI bot policies and read the Training setting. Disallow keeps mixed-use crawlers for search. Block or Block on pages with ads also stops Googlebot, Bingbot and Applebot.
- Check whether any per-crawler exception you rely on lives in a custom rule. If it does, Bot Preference Sync will not represent it.
One more thing worth knowing before anyone panics about search Console noise: Cloudflare's own documentation states that "Google Search Console may occasionally report Syntax not understood for Content Signals and newer directives in the robots.txt standard. However, we have observed no impact on crawling rates or SEO as a result of these reports." That is Cloudflare's observation, not Google's statement, and it is worth treating as a vendor claim rather than a settled fact.
India-specific considerations
Indian publishers and D2C brands sitting on Cloudflare's Free tier are affected by a detail that is easy to miss. Free-plan domains that have no robots.txt of their own and do not use the managed feature are served Cloudflare's Content Signals Policy text when a crawler requests the file. That policy defines the search, ai-input and ai-train categories and explicitly reserves rights under Article 4 of EU Directive 2019/790, but it expresses no preference of its own. A site owner in Gurugram or Bengaluru reading their own robots.txt may reasonably conclude they have opted out of training when they have not.
For teams whose Indian traffic depends on AI-assisted discovery rather than classical blue links, the practical order is: decide the Training position first, confirm the setting Cloudflare migrated you to on 15 September second, and only then decide whether to let Cloudflare own the file. robots.txt compliance remains voluntary in every case - Cloudflare's documentation says so directly: the file "expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level."
What is still unknown
Cloudflare's docs of 22 October 2026 show an example list of 8 crawlers, but not the generated user-agent list for any category. They give no refresh cadence for that list, which the August post only calls periodic, and no API or Terraform surface for Bot Preference Sync. There is no entry in the AI Crawl Control changelog as of 2 November 2026. The docs example also uses the marker Cloudflare Managed content rather than the marker in the August post, so check for both.
The safe operational position is unchanged: keep ownership of the file you can diff. If you have a hand-maintained robots.txt with per-crawler exceptions, leave the sync off unless its output matches your rules, and check which setting Cloudflare migrated you to. The real cost here is usually the audit after the fact, not the toggle.
FAQ
How eCorpIT can help
eCorpIT runs technical AI-search audits for teams whose discovery now depends on answer engines as much as on classical search results, and this is exactly the class of change those audits catch: a vendor default that rewrites a file nobody on the team owns. Our SEO, AEO and GEO audit service checks the served robots.txt, the content-signal state, the edge rules behind it, and whether the two agree. eCorpIT is CMMI Level 5, MSME certified and ISO 27001:2022 certified. Tell us your Cloudflare plan tier and current Training setting at /contact-us/ and we will tell you what the 15 September change means for your site.
References
- Cloudflare Blog, Say it once: introducing Bot Preference SynC, 21 August 2026.
- Cloudflare Blog, Your site, your rules: new AI traffic options for all customers, 1 July 2026.
- Cloudflare Docs, robots.txt setting, last updated 22 October 2026.
- Cloudflare Docs, AI Crawl Control overview.
- Cloudflare Docs, Directives (formerly Robots.txt), last updated 23 April 2026.
- Cloudflare Docs, Manage AI crawlers.
- Cloudflare Docs, BotBase.
- Cloudflare Docs, Verified bots.
- Cloudflare Docs, AI Crawl Control changelog.
- Cloudflare Radar, AI Insights and AI bot transparency.
- Cloudflare Radar, public bots directory.
- Cloudflare Docs, AI Crawl Control with Cloudflare WAF.
Last updated: 2 November 2026.
