Tag
Cloudflare's Bot Preference Sync, announced 21 August 2026, writes robots.txt from your Search, Agent and Training settings. It is category-wide only and replaced Managed Robots.txt on 15 September.
Manu Shukla 
Crawl-to-refer ratios move fast. Anthropic fell from 38,744:1 in July 2025 to about 517:1 for 25 to 31 October 2026, while OpenAI swung from 251:1 in July to about 1,900:1. Dates matter.
Manu Shukla 
Since September 15, 2026 Cloudflare's Disallow AI Training states a no-training preference and keeps Googlebot for search, while Block now stops it. How to set the three classes deliberately.
Manu Shukla 