Cloudflare Automates robots.txt Sync for AI Bot Policies
August 24, 2026
Cloudflare now automatically synchronizes AI bot preferences across Search, Agent, and Training categories within robots.txt for all user plans. This simplifies the enforcement of crawler permissions across different AI categories.
HOW THIS AFFECTS YOU
●
builderYou can manage bot access more efficiently across different AI use cases without manual file updates.
●
policyThis provides a more standardized way to signal intent regarding model training and agent crawling.