Cloudflare replaces its blanket AI bot block with granular controls for search, training, and agent crawlers

Cloudflare has announced a significant shift in how it handles AI bot traffic on its platform. Instead of blocking all AI bots indiscriminately, the company is rolling out granular controls that let website owners manage three distinct categories of AI crawlers: Search, Training, and Agent bots. This change aims to provide more flexibility for site operators who want to balance the benefits of AI-driven indexing and training with the need to protect their content and user experience.

The new controls will be available to all Cloudflare customers, with a phased rollout beginning September 15, 2026. Notably, Training and Agent bots will be blocked by default on pages that rely on advertising revenue, addressing concerns about data scraping and potential revenue loss. Search bots, however, will remain unaffected, ensuring that search engine visibility is preserved.

This move reflects growing tensions between AI companies and website operators over data access and content scraping. Cloudflare's decision to differentiate between bot types could set a new standard for how the industry handles AI-driven traffic, balancing innovation with fair usage policies.

Source: Cloudflare replaces its blanket AI bot block with granular controls for search, training, and agent crawlers. Read the full piece at the source.