Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Cloudflare requires AI companies to separate AI-focused crawlers from search crawlers by September 15 or face default blocking on publisher sites.
- Cloudflare’s policy requires AI companies to separate AI-focused crawlers from search crawlers by September 15, 2026.
- Failure to comply risks default blocking on publisher sites protected by Cloudflare.
- The move addresses publisher concerns over uncompensated AI model training data scraping.
- AI firms may face higher operational costs or need to negotiate licensing agreements.
Cloudflare has introduced a policy requiring AI companies to distinguish between web crawlers used for search and those used for AI training or agent interactions. By September 15, 2026, AI firms must either separate these crawlers or risk being blocked by default on publisher sites protected by Cloudflare. The move aims to address growing concerns over AI companies scraping publisher content without compensation, potentially disrupting access to high-quality data for AI training.
Publishers have long criticized AI firms for using their content to train models without permission or payment. Cloudflare’s policy shifts the burden to AI companies, forcing them to either negotiate licensing agreements or face technical barriers. This could significantly impact AI model training pipelines, particularly for companies relying on large-scale web scraping.
The policy also introduces a compliance mechanism where Cloudflare will enforce the separation, reducing the need for publishers to manually block AI crawlers. This could streamline publisher-AI company interactions but may increase operational costs for AI firms.
Source: Cloudflare’s new policy pushes AI companies to pay for publishers’ content. Read the full piece at the source.
AI developers may need to adjust crawler configurations or seek licensing agreements.
Companies relying on web-scraped data for AI training could face disruptions or increased costs.
Investors in AI firms may reassess risks related to data access and compliance.
Publishers gain leverage in negotiating compensation for AI training data usage.
- web crawler
- Automated bot that systematically browses the web to collect data.
- AI agent
- Software entity that performs tasks autonomously using AI models.