Cloudflare plans to automatically block mixed-use web crawlers that index websites for search engines while also serving as AI agents and trainers. The company previously offered customers the option to prevent these crawlers from scraping their sites for AI chatbots but is now adopting a more defensive default position.
CEO Matthew Prince stated, “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.” He emphasized that the new tools and partnerships give website owners increased visibility and commercial opportunities while benefiting AI companies with transparent intent.
Starting September 15, 2026, new customers and new websites from existing subscribers will have settings that default to allow search but block training and agent use on pages containing ads. Mixed-use crawlers that do not allow site owners to control AI content use will also be blocked on ad pages by default. Free account users will automatically switch to these new defaults unless they opt-out before the deadline.
Cloudflare is updating its Pay Per Crawl feature, now named Pay Per Use, where site owners will receive compensation when their content is referenced by AI chatbots. The announcement includes partnerships with Ceramic.AI and You.com, with expectations of attracting further AI companies to adopt these changes.
The new policy is viewed as a strategic move to challenge Google, which has access to significantly more information than leading AI companies. Google’s main web crawler, Googlebot, effectively indexes sites and collects data for its AI models, complicating publishers’ options regarding content usage. Cloudflare’s policies aim to prompt Google and other firms with mixed-use crawlers to reconsider their strategies.
