OpenAI to adopt robots.txt signals for AI training preferences
October 30, 2026
OpenAI will honor shared robots.txt signals to allow website operators to communicate specific preferences regarding AI training data. This move standardizes how webmasters manage crawler access for model training purposes.
HOW THIS AFFECTS YOU
●
builderYou must account for these signals when designing web scrapers or data ingestion pipelines.
●
policyYou can now use standard web protocols to enforce data governance and copyright protections.