On this page · 14 sections
- What changed between July 2025 and October 2026
- Option 1: Cloudflare Pay Per Use
- Option 2: Cloudflare Pay Per Crawl
- Option 3: Block or disallow AI crawlers
- Option 4: Licensing deals and marketplaces
- Option 5: Do nothing
- Head-to-head comparison
- What the traffic and revenue evidence says
- Decision table: pick this if
- Worked example: a 50,000-page B2B archive
- India, EU, UK and US notes
- Implementation checklist
- How eCorpIT can help
- References
Summary. Cloudflare opened Pay Per Use in beta on 30 September 2026. An AI company defines a use it will pay for, such as returning an excerpt from your page, and sets a price. You accept or decline in the dashboard, the buyer self-reports each use, and Cloudflare bills the buyer and pays you monthly. Pay Per Crawl, launched on 1 July 2025 and still in closed beta, is a different bargain: a crawler pays your price per request, with a $0.001 minimum, or gets an HTTP 402 response. Cloudflare publishes no earnings or fee figures for either, so nobody can show you a revenue number yet. Blocking is easier to judge. Cloudflare's Disallow AI Training setting, added on 15 September 2026, leaves Googlebot, Bingbot and Applebot crawling for search. On 30 October 2026 Google, Microsoft, OpenAI and Anthropic backed one shared robots.txt training signal; Microsoft honours it from 1 February 2027 and Google plans support in early 2027. Our view for most sites in 2027: keep search and AI-answer crawlers, disallow training, and test Pay Per Use only if you already run on Cloudflare and publish content buyers want.
About this article. For publishers, SEO leads and site owners deciding what to do about AI crawlers, including Indian sites. We checked product status, dates and crawler behaviour against Cloudflare's documentation, blog and Radar, against Microsoft, Google, OpenAI, Anthropic, Apple, Perplexity and Meta pages, against the IETF drafts (vocabulary -08, Kelley -00) and the DPIIT working paper Part 1 of December 2025, on 1 November 2026. This is general information, not legal advice, and no vendor paid for inclusion. Most licensing figures are press-reported, and we say which. Researched with AI assistance and reviewed by our editorial team.
What changed between July 2025 and October 2026
Cloudflare split monetisation into two products, rewrote what Block means for crawlers that serve both search and training, and the largest AI companies agreed on a common training opt-out. The dates come from Cloudflare, Microsoft and Google.
| Date | What happened |
|---|---|
| 1 Jul 2025 | Pay Per Crawl enters private beta, using HTTP 402 Payment Required |
| 3 Feb 2026 | Microsoft announces its Publisher Content Marketplace pilot |
| 15 Sep 2026 | Disallow AI Training arrives; Block now also stops Googlebot, Bingbot and Applebot |
| 30 Sep 2026 | Pay Per Use and Monetization Gateway enter beta |
| 30 Oct 2026 | Shared robots.txt training token backed by Google, Microsoft, OpenAI and Anthropic |
| 1 Feb 2027 | Microsoft starts honouring new training and grounding preferences as it crawls |
| 1 Mar 2027 | Bing stops reading NoArchive and NoCache as AI preferences |
| Early 2027 | Google plans to support the shared training token |
Option 1: Cloudflare Pay Per Use
Pay Per Use charges for what an AI product does with your page, not for fetching it. Cloudflare describes four steps.
- The buyer defines the use and the price. Each AI company identifies its crawler through Verified Bots, defines the paid use, sets a price and connects a payment account. Cloudflare's examples: a search service paying when it returns an excerpt, and a shopping agent paying when a review shapes a recommendation, even if the shopper never reads it.
- You accept or decline. Offers appear in the dashboard showing the AI company, the use and the price. You can stop later, and no origin change is needed. Each programme's terms state what the buyer may do with the content, including any limits on training.
- The buyer reports each use. One line of JSON per use goes to a single API. Usage is self-reported. Programme terms require complete reporting, and Cloudflare checks that each use maps to an enrolled publisher.
- Cloudflare settles both sides. It charges the buyer and pays publishers monthly. The dashboard shows estimated earnings and reported uses by buyer and domain.
The post names no buyers, gives no price range, states no Cloudflare fee and sets out no eligibility rules or regions. Search Engine Journal reports that Cloudflare described the search API Ceramic as a partner in July. Countering on price is a stated goal, not a feature. The same day Cloudflare put Monetization Gateway into closed beta for APIs and tools, charging agents per request over the open x402 protocol, but only for eligible US-based sellers and buyers. A seller based in India cannot use that route yet.
Limits: beta, buyer-set prices, usage you cannot audit independently, and it runs from the Cloudflare dashboard.
Option 2: Cloudflare Pay Per Crawl
Pay Per Crawl charges for access. You set a price for the zone, and each crawler you mark Charge either carries payment intent in its request headers and gets HTTP 200, or receives HTTP 402 with the price. The documentation, last updated 28 July 2026, still lists the product as closed beta with a signup form for site owners.
The minimum is $0.001 per crawl. One default price applies to every crawler set to Charge, and a crawler pays again each time it recrawls a page. Error responses are not billed, and files such as /robots.txt and /sitemap.xml are always free. You can price dynamically by returning a crawler-price header from your origin, or use a Worker, and you can exempt URI patterns through Configuration Rules. A WAF or Bot Management block overrides the charge.
Payouts are monthly through a dedicated Stripe Connect account, subject to settlement periods and minimum thresholds, and your accrued balance is not visible in the dashboard today.
The weakness is one Cloudflare itself names: AI products fetch far more than they use, so per-crawl charging makes buyers pay before they know what they need, and many will refuse. A 402 that nobody pays is, in practice, a block with a price attached.
Option 3: Block or disallow AI crawlers
robots.txt is a request. You name each vendor's token. GPTBot is OpenAI's training crawler, ClaudeBot is Anthropic's, Google-Extended controls Gemini training and grounding, Applebot-Extended controls Apple's foundation-model training, Meta-ExternalAgent covers Meta training and indexing, Amazonbot may train Amazon models, and CCBot is Common Crawl. Search and answer crawlers use separate tokens at OpenAI and Anthropic, so you can refuse training and stay in their answers. Our companion guide on AIPREF and robots.txt AI rules lists every token and a ready-to-use file. From 2027 one shared token, __usage_ai-training, will also carry a training opt-out to each provider that adopts it.
The network is enforcement. Disallow AI Training publishes a no-training preference, keeps Applebot, Bingbot and Googlebot crawling for search, and blocks every other training crawler, including the training-only crawlers run by Amazon, Anthropic, Meta and OpenAI. It is available on all plans. Block now applies to mixed-use crawlers, so choosing it removes you from Google and Bing search.
For Google, Apple and Microsoft the fetch still happens, and the preference relies on the operator honouring it. For Bing, Cloudflare says the setting does not yet convey a no-training preference through robots.txt, so you still need the NOARCHIVE tag until Microsoft's support starts. Cloudflare's managed list, in documentation dated 22 October 2026, also shows no shared token yet, so add your own group; our note on how Cloudflare prepends robots.txt covers the merge.
robots.txt alone cannot stop a crawler that ignores it. TollBit data reported by PPC Land found about 15% of identified AI page fetchers on European sites reached disallowed URLs. OpenAI and Perplexity say robots.txt may not apply to their user-triggered fetchers, and Meta says the same of Meta-ExternalFetcher. Anthropic's page says its bots honour robots.txt.
The hardest case is Googlebot. It feeds Google Search, including AI Overviews and AI Mode, so blocking it removes you from Google entirely. Google points site owners to snippet directives such as nosnippet and max-snippet, plus the Search generative AI control in Search Console, for limiting AI features.
Option 4: Licensing deals and marketplaces
Direct deals pay the most, for the few who can get them. LLM Pulse's tracker, updated on 28 September 2026, labels most figures as press-reported. We checked two with the parties' own statements. Informa disclosed its Microsoft terms in a stock exchange notice, and Meta and News Corp confirmed their deal but not its value.
| Buyer and partner | Value | Status of figure |
|---|---|---|
| OpenAI and News Corp | $250 million over five years | Reported, not confirmed |
| Google and Reddit | $60 million a year | Reported; Reddit's IPO filing gave $203 million across data deals |
| Amazon and The New York Times | $20 million to $25 million a year | Reported, not confirmed |
| Meta and News Corp | Up to $50 million a year over three years | Value reported by the WSJ; deal confirmed |
| Microsoft and Informa | Initial data access fee of $10m+, then recurring payments for 2025 to 2027 | Disclosed by Informa |
These describe large catalogues. A mid-sized B2B blog has no comparable bargaining power, which is the gap marketplaces try to fill.
Microsoft's Publisher Content Marketplace (PCM), announced on 3 February 2026, lets publishers define licensing and usage terms while AI builders pay for content in specific grounding scenarios. Publishers are paid on delivered value and get usage-based reporting. Microsoft co-designed it with publishers including The Associated Press, Conde Nast and Vox Media, and said it was onboarding demand partners including Yahoo. A Microsoft vice president told Digiday in February that the pilot contracts were negotiated individually and that a standard contract was not final. It is a pilot with an interest form, not a self-serve product, and we found no later status update from Microsoft.
Really Simple Licensing (RSL) is a different tool: an open standard, at version 1.0 since December 2025, that declares machine-readable terms through a License: line in robots.txt. It states your terms. It does not bring a buyer, and we found no public commitment from OpenAI, Google or Anthropic to honour it.
One point helps licensors. The shared-token draft says private agreements may override it, and the AIPREF vocabulary draft says explicit legal agreements likely apply despite machine-readable preferences. You can publish a blanket opt-out and still sell a licence to a named buyer.
Option 5: Do nothing
Microsoft states that its training and grounding preferences default to Allow, and the shared-token draft treats a robots.txt without it as expressing no preference. You keep full visibility in search and AI answers and give training access away for free.
Cloudflare's August 2025 analysis shows what that access was mostly used for: over the 12 months to July 2025, 80% of AI crawling was for training, 18% for search and 2% for user actions. Since 15 September 2026, Cloudflare recommends Disallow AI Training to new domains that earn from ads, and Allow to those that do not.
Head-to-head comparison
| Factor | Pay Per Use | Pay Per Crawl | Block or disallow | Licensing | Do nothing |
|---|---|---|---|---|---|
| Status, Nov 2026 | Beta | Closed beta | Available now | Bilateral deals; PCM pilot | Default |
| Who sets the price | AI buyer | You, per zone or dynamically | Not applicable | Negotiated | Not applicable |
| What triggers payment | Each reported use | Each successful fetch | Nothing | Contract terms | Nothing |
| Published prices or fees | None | $0.001 minimum; no fee published | Free | Rarely disclosed | Free |
| Effect on search traffic | None stated | None if search bots are not charged | None if you disallow rather than block mixed-use crawlers | None | None |
| Effect on AI answer citations | Depends on buyer terms | Lost if a crawler refuses to pay | Kept if answer bots stay allowed | Set by contract | Kept |
| Needs Cloudflare | Yes | Yes | No for robots.txt; yes for its network rules | No | No |
What the traffic and revenue evidence says
The strongest public data on what AI crawlers give back is Cloudflare Radar's crawl-to-refer ratio: HTML pages crawled per referral. The first two columns are full-month Radar pulls published by TechnologyChecker, and they match our crawl-to-referral analysis. The third is Radar's own seven-day view, read on 1 November 2026.
| Platform | Jul 2025 | Jul 2026 | 25 to 31 Oct 2026 |
|---|---|---|---|
| Anthropic | 38,744 | 1,917 | 517 |
| OpenAI | 1,104 | 251 | About 1,900 |
| Perplexity | 195 | 289 | About 6,200 |
| Microsoft | 40.4 | 36.2 | 40.8 |
| 5.5 | 4.7 | 4.8 |
One week is noisy: OpenAI's Q3 2026 figure was 309 and Perplexity's was 1,083. Native apps often send no Referer header, so Cloudflare says the ratios may overstate the gap. The ratios also mix bots: that week GPTBot was 68.5% of OpenAI's crawling and ClaudeBot 81.4% of Anthropic's. Even so, AI platforms crawl far more per referral than Google does, and at OpenAI and Anthropic most of that crawl is the training bot, which is why we expect blocking training crawlers to cost most sites little referral traffic.
The real trade-off sits in AI answers. Cloudflare's September 2026 post, citing Pew Research Center and other studies, says more than half of consumers read AI summaries in search and are over 40% more likely to end their search afterwards, while visitors referred by AI search convert at three to over five times the rate of traditional search visitors. Microsoft adds that neither allowing training nor allowing grounding guarantees inclusion, traffic or payment.
On revenue, we found no per-use or per-crawl earnings data from Cloudflare, Microsoft or any participating publisher as of 1 November 2026.
Decision table: pick this if
| Pick | If |
|---|---|
| Disallow training, keep search and answer bots | You are an ad-funded or lead-generation site and want the default that protects traffic |
| Disallow plus Pay Per Use | You already proxy through Cloudflare and publish fresh news, research or trade content |
| Pay Per Crawl | You want bots to pay or stay out and accept that few may pay |
| Licensing or PCM | You own a large, distinctive catalogue or archive and can negotiate |
| Block everything except search | Your content is your product and AI answers replace the visit |
| Do nothing | You want maximum AI exposure and do not sell content |
Worked example: a 50,000-page B2B archive
This example uses Radar ratios, Cloudflare's minimum price and one labelled placeholder. Swap in your own numbers. Assume a B2B publisher on Cloudflare sees 300,000 requests a month from OpenAI crawlers and 300,000 from Anthropic crawlers.
- Referrals at July 2026 ratios: 300,000 ÷ 251 is about 1,195 visits from OpenAI products; 300,000 ÷ 1,917 is about 156 from Anthropic. Total: roughly 1,351 visits a month.
- Referrals at 25 to 31 October 2026 ratios: 300,000 ÷ 1,900 is about 158; 300,000 ÷ 517.2 is about 580. Total: roughly 738.
- Pay Per Crawl at the $0.001 minimum: 600,000 × $0.001 is $600 a month if both operators paid for every fetch. If neither pays, revenue is $0 and the crawlers are effectively blocked.
- Pay Per Use at a placeholder $0.002 per reported use: if one buyer reports 40,000 uses, that is $80 a month.
- Disallowing training: at Radar's October split, GPTBot and ClaudeBot are about 205,500 and 244,200 requests, roughly 75% of the 600,000. Search and user fetchers can stay allowed, so most referrals should survive.
The two referral totals differ by almost half, so measure your own site. Disallowing training removes the crawls least likely to send anyone back, which is why it is the low-risk default. Monetisation is an experiment on top.
India, EU, UK and US notes
India. A DPIIT committee's working paper of December 2025, One Nation One License One Payment, recommended by majority a mandatory blanket licence: developers could train on lawfully accessed works, rights holders could not withhold them, and those who register would receive royalties through a central body. Nasscom dissented in favour of a text-and-data-mining exception with machine-readable opt-outs. Comments closed on 6 February 2026, and Digital Watch, citing Hindustan Times, reported in October that no revised Part 1 has appeared. Separately, on 24 July 2026 the Delhi High Court refused ANI an interim injunction against OpenAI, holding on a preliminary view that storing its works for training falls within section 52(1)(a) of the Copyright Act. ANI has appealed, and the appeal is reported as listed for 8 December 2026. If the licence were adopted, a robots.txt opt-out would not stop training under Indian law. Neither question is settled.
EU. Google says the shared token is consistent with the EU Code of Practice for general-purpose AI model providers, which it describes as recognising robots.txt as a primary way to express machine-readable preferences.
UK. The CMA's publisher conduct requirement comes into force on 3 December 2026, with AI Overviews and AI Mode controls and engagement metrics for content shown to UK users. See our guide to CMA AI Overviews metrics.
US. We found no statutory licensing scheme for AI training. Microsoft notes its preferences create no payment arrangements, so payment comes from contracts or marketplaces.
Implementation checklist
- Pull 30 days of crawler data from AI Crawl Control or server logs, and separate training, search and user-fetch bots.
- Publish a robots.txt that disallows training tokens, leaves search and answer tokens allowed, and adds
__usage_ai-training.
- On Cloudflare, choose Disallow AI Training, not Block, unless you intend to leave Google and Bing search.
- If you use Bing's NoArchive or NoCache tags, publish the new preferences before 1 March 2027, and record the temporary grounding preference from 15 December 2026 if you withhold grounding.
- Decide which sections you would sell, and apply for Pay Per Use, Pay Per Crawl or Microsoft's PCM.
- Track AI referrals and conversions in analytics, so the trade-off is measured on your own site.
How eCorpIT can help
eCorpIT is a Gurugram-based technology organisation, founded in 2021, assessed at CMMI Level 5 and MSME certified, with senior-led engineering teams working across AWS, Microsoft and Google platforms. Our performance marketing services team can audit crawler logs, set robots.txt and Cloudflare rules that keep search visibility, and measure what AI referrals are worth. Talk to us at /contact-us/.
Last updated: 1 November 2026.
