On this page · 10 sections
Summary. Cloudflare published "Say it once: introducing Bot Preference SynC" at 23:19 UTC on 21 August 2026. The feature generates or rewrites your robots.txt from the Search, Agent and Training settings you already hold in the zone dashboard, and it is promised to every plan tier from Free to Enterprise "in the coming week". Three details in that post matter more than the headline. It works only on category-wide policy, so it will not read your per-crawler custom rules. It is on by default for all new customers. And as of this writing the documentation page it needs, at /ai-crawl-control/features/managed-robots-txt/, returns a 404, while the page that does exist - the legacy robots.txt setting doc, last updated 3 August 2026 - still describes a different mechanism that hard-writes Disallow: / for 8 named crawlers. If you run a site behind Cloudflare with a hand-maintained robots.txt, the sequencing here decides whether your file says what you think it says on 15 September 2026, when Cloudflare's new AI traffic defaults take effect.
What Cloudflare actually announced
The mechanism is narrow and worth stating exactly. Bot Preference Sync does not introduce new blocking. It takes the AI bot policy you have already configured at zone level - Allow, Block on pages that serve ads, or Block everywhere for Search and Agent, plus Disallow for Training - and reflects that policy as text inside robots.txt.
If your origin already serves a robots.txt, Cloudflare prepends its block ahead of your content rather than replacing it, wrapped in two comment markers:
# BEGIN Cloudflare Bot Preference Sync
User-agent: TrainingBot1
User-agent: TrainingBot2
User-agent: TrainingBot3
User-agent: MixedUseBot-Extended
Disallow: /
...
# END Cloudflare Bot Preference Sync
That example is Cloudflare's own, shortened and anonymised in the post. The real user-agent list is pulled from BotBase, Cloudflare's bot database launched on 1 July 2026, and Cloudflare says it will "periodically update the list of bots that is added to robots.txt when you choose to Block or Disallow a given category". You do not pin that list. Cloudflare does, and it changes underneath you.
The stated goal is to close the gap between what a site declares and what it enforces. As the announcement puts it: "When your stated preferences and your enforced rules disagree, some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules." That is a fair description of the problem. It is also the reason the custom-rules carve-out below matters.
The part the announcement does not lead with
Three limits sit further down the post, and each one changes an operational decision.
Category-wide only. Cloudflare states plainly that Bot Preference Sync "is designed to tackle policy decisions made category-wide rather than case-by-case, it will not directly read from individual custom rules with more complex logic." If you have negotiated an exception for one named crawler - a licensing deal, a partner index, an internal scraper - that exception lives in a custom rule and will not appear in the generated file. Your robots.txt will then advertise a stricter policy than your edge actually enforces, which is the exact mismatch the feature was built to remove. Cloudflare's answer is to turn the sync off and hand-maintain the file.
On by default for new customers. The post says Bot Preference Sync "will be on by default" for all new customers. Read alongside the next paragraph, that means the sync is on, not that blocks are on: for non-publisher new customers Cloudflare "will not have any blocks or disallows added by default". The two statements are consistent, but they read as contradictory on a first pass, and the practical effect is that a newly onboarded zone gets a Cloudflare-managed robots.txt file whether or not it has any AI policy set.
A new publisher default. At onboarding, a customer can select "I monetize from pages with ads on this domain", which sets Training to Disallow as the default. That is an opt-in checkbox with a policy consequence, changeable at any time.
The docs have not caught up
The AI Crawl Control documentation set does not contain a Bot Preference Sync page. Requesting /ai-crawl-control/features/managed-robots-txt/ returns Cloudflare's 404 template. The navigation lists Analyze AI traffic, Manage AI crawlers, and Directives - the tab formerly called Robots.txt - and nothing else in that area. The Directives page itself carries a "Last updated Apr 23, 2026" stamp and still points its managed-status card at the older bots/additional-configurations/managed-robots-txt/ page.
That older page, stamped 3 August 2026 in its own structured data, documents a materially different file. It writes a fixed content-signal line and a blanket disallow for a named list:
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
The full list in that document is 8 crawlers: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, and meta-externalagent. Turn on the legacy toggle today - Security Settings, filter by Bot traffic, "Set your preference to block training in robots.txt" - and that is what lands in your file. It is not the Search/Agent/Training model the 21 August post describes.
| Behaviour | Legacy managed robots.txt (doc updated 3 Aug 2026) | Bot Preference Sync (announced 21 Aug 2026) |
|---|---|---|
| Source of directives | Fixed list of 8 named crawlers | BotBase categories, periodically refreshed |
| Policy model | Block training, one setting | Search, Agent, Training set separately |
| Reads your custom rules | No | No - category-wide only |
| Placement in your file | Prepended above existing content | Prepended above existing content |
| Documentation | Live page, all plans | No page; 404 at the expected path |
Why the 15 September date is the real deadline
Bot Preference Sync publishes a preference. It does not decide who gets blocked at the edge, and the edge rules change on a fixed date.
In "Your site, your rules: new AI traffic options for all customers", published 1 July 2026, Cloudflare wrote: "On September 15, 2026, we'll be setting new defaults for each of these three classifications." For all new domains onboarding to Cloudflare, Training and Agent are blocked by default on pages that display ads, while Search stays allowed. The same post carries the sentence that costs the most money if you miss it: multi-purpose crawlers will be "allowed/blocked according to all of their behaviors", with the most restrictive applicable rule winning, so "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service)."
Read that twice if you turned on "Block AI bots" in 2025 and have not looked since. Cloudflare's opt-out is a setting in the zone's Security settings, available "any time leading up to September 15", which confirms you want no change to Training crawlers that also crawl for Search.
The 21 August post adds a partial escape hatch on the preference side. The Training Disallow option writes a "no training" preference rather than a hard block, and Cloudflare says cooperating mixed-use crawlers that meet four transparency requirements keep search access. Those four requirements, verbatim from the post, are that the bot respects a "no training" preference in robots.txt by any mechanism; gives site owners a way to opt out of AI summaries; provides URL-level visibility into which pages were made available for training plus metrics on search results; and can show publicly that disallowing training does not hurt traditional search results. Operators meeting them are tracked in the AI bot transparency section on Cloudflare Radar. Operators that do not "are still blocked when you disallow training".
So the preference layer and the enforcement layer resolve differently, on different dates, using different lists. That is the whole problem for anyone measuring AI search visibility.
How to tell whether this is you
Four checks, none of which take longer than a few minutes.
- Fetch your own
robots.txtand look for# BEGIN Cloudflare Managed contentor# BEGIN Cloudflare Bot Preference Sync. Either marker means Cloudflare is writing part of your file.
- Look for a
Content-signal:line. Cloudflare's managed content setssearch=yes, ai-train=no, use=reference. Theusefield is an optional extension Cloudflare describes as under test, with three values:use=immediate(interact, store and reuse nothing),use=reference(index, excerpt, and link back), anduse=full(summarize and reproduce).
- Open Security Settings and check whether "Block AI bots" or the legacy training-preference toggle is on. If it is, the 15 September multi-purpose rule applies to you.
- Check whether any per-crawler exception you rely on lives in a custom rule. If it does, Bot Preference Sync will not represent it.
One more thing worth knowing before anyone panics about search Console noise: Cloudflare's own documentation states that "Google Search Console may occasionally report Syntax not understood for Content Signals and newer directives in the robots.txt standard. However, we have observed no impact on crawling rates or SEO as a result of these reports." That is Cloudflare's observation, not Google's statement, and it is worth treating as a vendor claim rather than a settled fact.
India-specific considerations
Indian publishers and D2C brands sitting on Cloudflare's Free tier are affected by a detail that is easy to miss. Free-plan domains that have no robots.txt of their own and do not use the managed feature are served Cloudflare's Content Signals Policy text when a crawler requests the file. That policy defines the search, ai-input and ai-train categories and explicitly reserves rights under Article 4 of EU Directive 2019/790, but it expresses no preference of its own. A site owner in Gurugram or Bengaluru reading their own robots.txt may reasonably conclude they have opted out of training when they have not.
For teams whose Indian traffic depends on AI-assisted discovery rather than classical blue links, the practical order is: decide the Training position first, confirm the 15 September opt-out state second, and only then decide whether to let Cloudflare own the file. robots.txt compliance remains voluntary in every case - Cloudflare's documentation says so directly: the file "expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level."
What is still unknown
Cloudflare has not published the generated user-agent list for any category, the refresh cadence for that list beyond "periodically", the API or Terraform surface for Bot Preference Sync, or a dated availability commitment beyond "in the coming week". There is no changelog entry in the AI Crawl Control changelog as of this writing. Until the documentation page exists, the only authoritative description of the feature is a blog post, and blog posts are not a contract.
The safe operational position for the next fortnight is unchanged: keep ownership of the file you can diff. If you have a hand-maintained robots.txt with per-crawler exceptions, leave the sync off until the docs land, and re-check your setting on 14 September. The real cost here is usually the audit after the fact, not the toggle.
FAQ
How eCorpIT can help
eCorpIT runs technical AI-search audits for teams whose discovery now depends on answer engines as much as on classical search results, and this is exactly the class of change those audits catch: a vendor default that rewrites a file nobody on the team owns. Our SEO, AEO and GEO audit service checks the served robots.txt, the content-signal state, the edge rules behind it, and whether the two agree. eCorpIT is CMMI Level 5, MSME certified and ISO 27001:2022 certified. Tell us your Cloudflare plan tier and current Training setting at /contact-us/ and we will tell you what changes on 15 September.
References
- Cloudflare Blog, Say it once: introducing Bot Preference SynC, 21 August 2026.
- Cloudflare Blog, Your site, your rules: new AI traffic options for all customers, 1 July 2026.
- Cloudflare Docs, robots.txt setting, last updated 3 August 2026.
- Cloudflare Docs, AI Crawl Control overview.
- Cloudflare Docs, Directives (formerly Robots.txt), last updated 23 April 2026.
- Cloudflare Docs, Manage AI crawlers.
- Cloudflare Docs, BotBase.
- Cloudflare Docs, Verified bots.
- Cloudflare Docs, AI Crawl Control changelog.
- Cloudflare Radar, AI Insights and AI bot transparency.
- Cloudflare Radar, public bots directory.
- Cloudflare Docs, AI Crawl Control with Cloudflare WAF.
Last updated: 22 August 2026.
