3 articles tagged
Cloudflare's Bot Preference Sync, announced 21 August 2026, writes robots.txt from your Search, Agent and Training settings. It is category-wide only and its documentation page returns a 404.
The crawl-to-refer ratios circulating in 2026 are a year out of date. Anthropic fell from 38,744:1 to 1,917:1 in twelve months, OpenAI from 1,104:1 to 251:1, and Cloudflare says every figure overstates the case.
From September 15, 2026 Cloudflare blocks Training and Agent crawlers by default on ad pages, and multi-purpose bots like Googlebot get caught in between. Here is how to set the three classes deliberately.
