Since September 15, 2026, Cloudflare blocks crawlers it classifies as Training or Agent on pages that display ads for new domains by default, while Search stays allowed. Cloudflare can also add a managed section to the robots.txt it serves, so the file crawlers read can differ from the file you deploy. Check the served robots.txt for the Cloudflare Managed content section, then review the AI bot policies in Cloudflare’s security settings, because edge blocking is not visible in robots.txt.
If your site runs behind Cloudflare and your AI crawler setup no longer matches what you remember choosing, this page shows where each setting lives and how to check it.
What changed on September 15, 2026
Cloudflare groups AI crawler traffic into three behaviors and lets every customer, including Free plan customers, set a policy for each:
| Behavior | What Cloudflare means by it |
|---|---|
| Search | Collects or indexes your content so it can answer questions about it later. |
| Agent | Acts in real time on a person’s behalf, such as chat fetch bots and browser-use agents. |
| Training | Crawls your content to train or fine-tune a model. |
Each behavior can be set to Block on all pages, Block on pages with ads, or Allow. Training also has a Disallow option, which works through robots.txt directives (see below).
Cloudflare’s documentation states that, from September 15, 2026:
- New domains onboarding to Cloudflare get Training and Agent set to Block on pages with ads, with Search allowed. Cloudflare detects ad-carrying pages automatically.
- Crawlers that combine Search and Training are also blocked by any configuration that blocks AI training, including the older Block AI bots toggle, which was deprecated on the same date.
- All customers could opt out of the new defaults before that date, so check your own zone rather than assuming which settings it has.
Cloudflare’s robots.txt documentation also says that when Training is set to Disallow with Bot Preference Sync turned on, crawlers that combine Search and Training are excluded from the edge block. Block and Disallow therefore treat these combined crawlers differently. Check which one your zone uses.
Two different mechanisms
Cloudflare can act on AI crawlers in two separate ways. Only one of them is visible from outside.
| Mechanism | Where it acts | Visible in your robots.txt? | How to check it |
|---|---|---|---|
| Edge blocking from AI bot policies | Cloudflare’s network, on each request | No | Cloudflare dashboard → Security Settings → Configure AI bot policies |
| Managed robots.txt from Bot Preference Sync | The robots.txt Cloudflare serves for your domain | Yes, between fixed comment markers | Fetch /robots.txt from the live domain, or run the robots.txt AI crawler checker |
Edge blocking stops a request before it reaches your site, and nothing in your robots.txt reveals it. The managed robots.txt only states preferences. Crawlers that follow robots.txt read them, and the robots.txt standard does not make them enforceable.
How Cloudflare changes your robots.txt
With Bot Preference Sync turned on, Cloudflare generates robots.txt directives whenever a policy is set to Block or Block on pages with ads, or Training is set to Disallow. It places them in a marked section:
# BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# END Cloudflare Managed ContentThe example above is shortened. Cloudflare’s version lists more crawlers, and a Content Signals Policy comment block sits above the section.
How the section reaches your file depends on your origin:
- Your origin serves a robots.txt: Cloudflare puts its section first and serves the combined result. The file in your repository or CMS does not change.
- Your origin has no robots.txt: Cloudflare creates one and serves it for you.
- Free plan, no robots.txt of your own, managed robots.txt off: Cloudflare serves the Content Signals Policy text, which defines the signals but states no preference.
Under the robots.txt standard, groups for the same crawler token are combined. A Disallow: / for GPTBot in the managed section therefore applies even if your own rules never mention GPTBot.
How to check your site
- Check the served file. Run the robots.txt AI crawler checker on your domain. If it finds the Cloudflare Managed content section, it shows the line range, the crawler groups inside it, and any content signal. It also marks crawlers that the section blocks or names as Added by Cloudflare.
- Compare it with the file you deploy. Rules in the managed section are not in your repository or CMS. A robots.txt edit cannot remove them.
- Compare the result with your goal. In the checker, pick a goal such as Search visibility, training opt-out. Conflicts that come from the managed section are labelled. Their fix is in Cloudflare’s settings, not in your file.
- Review the AI bot policies. In the Cloudflare dashboard, open Security Settings, filter by Bot traffic, and select Configure AI bot policies. Note what Search, Agent, and Training are set to, and whether Bot Preference Sync is on.
- Remember the edge. The robots.txt check does not show whether a crawler is blocked at Cloudflare’s network edge. The complete audit can request selected paths as a generic automated client, but that request is not a verified provider crawler, so it cannot reproduce how Cloudflare classifies a real one. The dashboard is the place to confirm edge blocking.
- Check again after any change. Run the checker again and confirm the rules and signals match what you chose.
What content signals mean
A content signal line such as Content-signal: search=yes, ai-train=no states how collected content may be used:
search: building a search index and showing links and short excerpts, not AI-generated summaries;ai-input: putting content into an AI model to generate answers in real time, for example retrieval or grounding;ai-train: training or fine-tuning a model.
yes grants that use, no withholds it, and a missing signal neither grants nor refuses it. Content signals are preferences, not access controls. Crawler operators decide whether to support them, so check each provider’s own documentation before relying on them. Cloudflare notes that Google Search Console may report Syntax not understood for these lines, and says it has seen no effect on crawling as a result.
What this check cannot tell you
- Whether a specific provider visited, indexed, cited, or recommended your pages.
- Whether Cloudflare blocked a particular request at the edge.
- Whether a crawler follows robots.txt or content signals. Some user-triggered fetchers are documented as not following robots.txt.
- How rules apply to paths other than the one tested. The checker evaluates the root path.
Use the crawler registry to see what each provider documents about its tokens, and the guide on allowing AI search while blocking model training if you want to set that policy in your own file.