Starting September 15, 2026, anyone with a site on Cloudflare will see how AI crawlers can access their pages change without doing anything — at least on pages carrying ads.
The new AI bot classification
Cloudflare has defined three categories for AI bots:
- Search — crawlers that index content to answer questions later (e.g. AI search crawlers). For these, you'd expect return traffic or other forms of fair compensation
- Agent — real-time automated activity on a person's behalf: bots fetching pages for a conversational assistant, agents browsing the web autonomously
- Training — crawlers that collect content to train or fine-tune an AI model
What changes in practice
Starting September 15, on pages that host advertising, Cloudflare will default-block bots classified as Training or Agent. Search bots remain allowed.
The logic: search crawlers, in theory, drive return traffic to the site, while training crawlers just collect content without giving anything back to the publisher — a model that, per Cloudflare, is no longer sustainable at scale for ad-supported publishers.
Who it applies to
The change is automatic for:
- All new Cloudflare customers
- New sites created by existing customers
- All free-tier users, including already-active ones
Sites already active on paid plans are not changed automatically — they stay on the previous configuration unless manually adjusted.
How to opt out (if needed)
Anyone who doesn't want this default behavior can turn it off from their security settings at any time before the deadline, explicitly confirming they don't want to restrict access to Training crawlers that also perform Search activity.
Why it matters for SEO and AI visibility
This change lands during a period of growing focus on visibility in AI answer engines (AI Overviews, ChatGPT, Perplexity) — the same ground where AEO/GEO, increasingly discussed, gets played out.
Default-blocking training crawlers isn't a neutral choice: it means consciously deciding whether your content should feed third-party AI models, with direct implications for how — and whether — that content later gets cited by those same AI tools. It's worth reviewing your settings before September 15, not leaving it to chance.
It's no longer just a robots.txt question. Cloudflare is building a granular control layer over AI traffic that effectively redefines who can read a site and under what terms — and it's worth knowing where you stand before the default changes on its own.