#AI crawlers

2 articles tagged

Crawl-to-refer ratio chart: Anthropic 1,917 to 1, OpenAI 251 to 1, Google 4.7 to 1 in July 2026

SEO & AI Search · Aug 6, 2026 · 14 min read

Crawl-to-refer ratios in July 2026: Anthropic 1,917:1, OpenAI 251:1

The crawl-to-refer ratios circulating in 2026 are a year out of date. Anthropic fell from 38,744:1 to 1,917:1 in twelve months, OpenAI from 1,104:1 to 251:1, and Cloudflare says every figure overstates the case.

Manu Shukla

#AI crawlers #GEO #cloudflare-radar

Editorial graphic on Cloudflare's Search, Agent and Training AI crawler classes with a Sept 15, 2026 deadline

SEO & AI Search · Jul 23, 2026 · 14 min read

September 15, 2026: Cloudflare's crawler split and how to configure Search, Agent and Training

From September 15, 2026 Cloudflare blocks Training and Agent crawlers by default on ad pages, and multi-purpose bots like Googlebot get caught in between. Here is how to set the three classes deliberately.

Manu Shukla

#cloudflare #AI crawlers #GEO