Dataconomy
  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
Subscribe
No Result
View All Result
  • AI
  • Tech
  • Cybersecurity
  • Finance
  • DeFi & Blockchain
  • Startups
  • Gaming
Dataconomy
  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
Subscribe
No Result
View All Result
Dataconomy
No Result
View All Result

New Cloudflare tool lets sites charge AI crawlers

Website owners can now set micropayment rates for each AI crawl or choose to block or allow access for free.

byKerem Gülen
July 1, 2025
in Artificial Intelligence, News
Home News Artificial Intelligence

Cloudflare announced the launch of “Pay per Crawl,” a private beta marketplace enabling website owners to charge AI crawlers for content access, fulfilling a vision articulated by CEO Matthew Prince in 2024.

Cloudflare has, over the past year, introduced various tools designed to manage AI crawler interactions for publishers. These include a one-click solution for blocking all AI bots and a dashboard providing visibility into how AI crawlers are accessing their sites. Cloudflare CEO Matthew Prince stated in a 2024 interview that these products were foundational steps toward creating a marketplace where publishers could monetize their content by distributing it to AI companies.

Cloudflare launched this marketplace, named “Pay per Crawl,” as a private beta experiment. Within this experiment, website owners can establish a specific micropayment rate for each “crawl” of their site by individual AI crawlers. Alternatively, website owners retain the option to permit AI crawlers to scrape their content without charge or to block them entirely. Cloudflare asserts that its tools provide website owners with the ability to discern whether crawlers are accessing their site for AI training data, for inclusion in AI search responses, or for other purposes.

Stay Ahead of the Curve!

Don't miss out on the latest insights, trends, and analysis in the world of data, technology, and startups. Subscribe to our newsletter and get exclusive content delivered straight to your inbox.

Cloudflare additionally announced that new websites configured with Cloudflare will, by default, block all AI crawlers. This modification requires site owners to explicitly grant permission for specific AI crawlers to access their content, a change Cloudflare describes as providing “the default of control” to every new domain. Several prominent publishers, including Conde Nast, TIME, The Associated Press, The Atlantic, ADWEEK, and Fortune, have committed to blocking AI crawlers by default in alignment with Cloudflare’s objective of promoting a “permission-based approach to crawling.”


Cloudflare fights AI scrapers with a maze of useless content


Data collected by Cloudflare in June 2025 indicates a disparity in crawling frequency between AI companies and traditional search engines. Cloudflare reported that Google’s crawler accessed websites 14 times for every referral it provided. In contrast, OpenAI’s crawler accessed websites 17,000 times for each referral, while Anthropic’s crawler accessed websites 73,000 times per referral. Cloudflare envisions “Pay per Crawl” potentially evolving into an “agentic paywall.”

This concept could involve AI agents, acting on behalf of users, being allocated a budget to acquire relevant content. Cloudflare stated in a blog post, “What if an agentic paywall could operate at the network edge, entirely programmatically? Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content.”

Participation in Cloudflare’s experimental marketplace necessitates that both AI companies and publishers maintain Cloudflare accounts. Within these accounts, both parties can establish rates for the purchase and sale of a “crawl” of the publisher’s content. Cloudflare functions as an intermediary in these transactions, charging the AI company and subsequently distributing the earnings to the publisher. Ripley Park, a Cloudflare spokesperson, confirmed that “Pay per Crawl” does not currently involve stablecoins or cryptocurrency.


Featured image credit

Tags: AIcloudflareFeatured

Related Posts

Asus ROG Ally, Ally X preorders open; ships October 16

Asus ROG Ally, Ally X preorders open; ships October 16

September 26, 2025
OpenAI: GDPval framework tests AI on real-world jobs

OpenAI: GDPval framework tests AI on real-world jobs

September 26, 2025
Hugging Face: AI video energy use scales non-linearly

Hugging Face: AI video energy use scales non-linearly

September 26, 2025
Apple Wallet digital ID expands to North Dakota

Apple Wallet digital ID expands to North Dakota

September 26, 2025
Salesforce Agentforce hit by Noma “ForcedLeak” exploit

Salesforce Agentforce hit by Noma “ForcedLeak” exploit

September 26, 2025
BetterPic: The industry-leading AI headshot generator

BetterPic: The industry-leading AI headshot generator

September 26, 2025

LATEST NEWS

Asus ROG Ally, Ally X preorders open; ships October 16

OpenAI: GDPval framework tests AI on real-world jobs

Hugging Face: AI video energy use scales non-linearly

Apple Wallet digital ID expands to North Dakota

Salesforce Agentforce hit by Noma “ForcedLeak” exploit

BetterPic: The industry-leading AI headshot generator

Dataconomy

COPYRIGHT © DATACONOMY MEDIA GMBH, ALL RIGHTS RESERVED.

  • About
  • Imprint
  • Contact
  • Legal & Privacy

Follow Us

  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
No Result
View All Result
Subscribe

This website uses cookies. By continuing to use this website you are giving consent to cookies being used. Visit our Privacy Policy.