Cloudflare has announced new classifications, enhanced analytics, and industry-defining commercial partnerships that bring together site owners and transparent AI companies so that the agentic Internet can flourish. Cloudflare’s new tools and integrations help site owners and AI companies optimise for discoverability, efficiency, and monetisation. By establishing these new rails, Cloudflare is helping provide the foundation for a healthy, collaborative agentic economy that benefits society, site owners, and AI companies.
The fundamental nature of Internet traffic has shifted. Today, automated agents and bots drive more than half of all web requests. AI is becoming the new interface for accessing information and conducting commerce online, changing the rules for how content is discovered. Most website owners want their content in AI systems. But for businesses built on advertising or subscriptions, they are threatened by AI systems training on their content without compensation. We believe that if you’re a business that wants your content in AI systems, we should make it as easy and efficient as possible; but if you’re a business where you do not, then you should have the tools to restrict AI’s access. In response, Cloudflare is testing new default classifications, delivering deeper insights to customers, making AI search faster, and ensuring creators are compensated when their content powers an answer. The result is a level playing field. It optimises crawling efficiency, protects intellectual property, and allows creators and AI to thrive in the agentic economy.
“Last year we provided site owners with transparency and control over what bots access their content, and we are thrilled with the benefits it has had to the ecosystem,” said Matthew Prince, co-founder and CEO of Cloudflare. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge. Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training.”
After thousands of conversations with AI companies and site owners, one thing is clear: most site owners want to be discoverable in AI. However, for those who monetise via ads and/or subscriptions, maximising discovery shouldn’t mean giving away intellectual property for free. Transparent AI companies have enabled this by clearly separating and labelling their bots by intent. However, mixed crawler bots disadvantage both site owners and transparent AI companies by blending search, agent use, and training. This forces content owners to choose between being discoverable in an agentic era and giving away their most valuable content without compensation. Today, the largest search engine has access to about 2X more information than leading AI companies because they make it difficult for customers to remain discoverable without also being used for AI.
Over the next two months, Cloudflare will engage with the ecosystem, listen to feedback, and run tests to finalize new defaults and classifications with a deadline of September 15, 2026.
For new customers and for new sites for existing customers, on September 15, 2026, the defaults will be set to allow for search but block training and agent use for pages with ads. Mixed crawlers that do not give site owners the ability to choose between search, agent use, and training, will be blocked on all pages with ads. At any time these settings can be changed by customers in their dashboard.
On September 15, 2026 these changes will also be made for all existing free customers that have not changed their settings by September 15, 2026 in their dashboard. These changes aim to level the playing field between AI companies, who are redefining what it means to be discoverable in an agentic era in an equitable and transparent way, and legacy search companies.
