\n<\/p>\n
Cloudflare has just issued the AI industry a new deadline to separate the web crawlers used for traditional search purposes, like Google Search, from those used for AI agents and training. Starting on September 15, 2026, Cloudflare\u2019s default settings will block \u201cmixed-use\u201d crawlers from any pages that host ads, the company announced on Wednesday.<\/p>\n
That means that the crawlers that blend search, agent use, and training will be blocked from crawling these sites by default, unless the site owner adjusts the settings otherwise. These changes to the defaults will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers, the company says.<\/p>\n
The move could impact how AI model providers are able to access web content for training purposes and to help power their agentic services. <\/p>\n
Cloudflare points out that most website owners want their content to be discoverable via search and often through AI services as well, but they want protections against having their intellectual property given away for free.<\/p>\n
Cloudflare specifically calls out the \u201cworld\u2019s largest search engine\u201d (clearly a Google reference!) as having access to about \u201c2x more information\u201d than other AI companies because the search giant makes it difficult for customers to remain discoverable without being used for AI. <\/p>\n
Google has pushed back against this generalization in the past, noting that it provides a bot called Google Extended<\/a> that lets site owners opt out of having their content used for training and AI products and services like Gemini Apps and Vertex API. Its use doesn\u2019t impact a site\u2019s inclusion in Google Search. However, the tech giant\u2019s flagship Googlebot crawls for Search, including AI features like AI Overviews and AI Mode.<\/p>\n \u201cNow that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,\u201d said Cloudflare co-founder and CEO Matthew Prince in his announcement of the news, referring to the recent milestone where bots surpassed human traffic online<\/a> for the first time. That shift was not expected to occur until next year.<\/p>\n \u201cCloudflare\u2019s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training,\u201d Prince said.<\/p>\n While Cloudflare offers a number of products to help users launch their own AI systems<\/a>, the company has also released a range of tools to give publishers more control over their content in the AI era. In recent years, Cloudflare launched tools to combat AI bots<\/a>, including a marketplace that lets websites charge AI bots for scraping<\/a>, dubbed Pay Per Crawl.<\/p>\n The latter is now also evolving into \u201cPay Per Use,\u201d the company said, which will allow publishers to charge AI companies when their content creates value, not just when it\u2019s fetched. <\/p>\n The change could also help conserve publishers\u2019 bandwidth and compute resources for AI model providers, as Cloudflare\u2019s data suggested that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages. <\/p>\n To put this into action, Cloudflare is initially working with two partners, Ceramic.ai and You.com. When a publisher opts in, they\u2019re paid when their content appears in Ceramic\u2019s AI search results or when You.com accesses a piece of their premium content. <\/p>\n Other AI companies can customize this model for how they work, Cloudflare says. <\/p>\n<\/div>\n When you purchase through links in our articles, we may earn a small commission<\/a>. This doesn\u2019t affect our editorial independence.<\/em><\/p>\n