Cloudflare separates AI training refusal from search crawler blocking

Cloudflare has introduced a control for publishers who want ordinary search indexing without consenting to model training through the same mixed-use crawler. Disallow AI Training publishes a no-training preference while leaving designated crawlers able to reach the site for search. A network block would refuse access to both uses when they share one crawler identity.

In its 15 September announcement, published at 13:00 UTC, Cloudflare also changed what Block means. Blocking Training now includes Applebot, Bingbot and Googlebot; their ordinary search access is blocked too. Disallow gives operators a way to state a use preference and retain that access. The preference cannot guarantee that content will never enter a model or an AI answer.

The choice changes because one crawler can have two uses

Cloudflare classifies crawler activity as Search, Training or Agent. Search builds an index; Training feeds model development; Agent refers to a user-directed fetch or browser-use visit. The announcement addresses Search and Training when one crawler combines them. Cloudflare did not add a Disallow option for Agent traffic, citing the absence of an established preference directive comparable to a training opt-out.

Before this change, Cloudflare’s July AI traffic options planned to subject mixed-use crawlers to the most restrictive applicable behaviour setting from 15 September. If a site blocked training, that approach could also stop a shared search crawler. The new arrangement distinguishes refusal of use from refusal of access. Under Disallow AI Training, Cloudflare says its Bot Preference Sync publishes applicable no-training directives in robots.txt, allows designated “Accountable” mixed-use crawlers for search, and blocks other training crawlers. Cloudflare’s designation includes operators with current controls or a commitment to provide them; it should not be read as an independent audit of every downstream use.

Choosing Block takes the access route instead. Cloudflare says both Block and Block on pages with ads now apply to mixed-use crawlers. A full Training Block stops Applebot, Bingbot and Googlebot from reaching the site, including for ordinary search. The ads-only setting blocks them on pages Cloudflare detects as serving ads, so those pages also lose crawler access. Search and Agent controls, another firewall rule or a manually maintained robots policy can impose additional limits. A Training choice alone does not override those other settings.

There is a technical boundary between those routes. The Robots Exclusion Protocol defines crawler-access preferences in robots.txt; RFC 9309 says it is not access authorisation or a substitute for application security. A published no-training rule cannot identify the requestor, verify what a crawler later does with data, or stop a client that ignores it. Cloudflare can classify and block traffic at its edge. An operator needs to distinguish that access refusal from a use preference honoured by a cooperating crawler.

Migration preserves the old effect, but the labels have changed

An existing AI blocking selection may have changed its label. Cloudflare says domains using granular Search, Training and Agent controls retain the practical effect of their selections: previous Training Block and Block on pages with ads choices migrate to Disallow AI Training. Search and Agent selections carry over. The mapping avoids suddenly removing search access from sites whose old training rule did not block mixed-use crawlers. An operator who now wants to stop those crawlers needs to select the new Block deliberately.

For domains that used only the legacy Block AI Bots switch, Cloudflare’s transition table maps the old Block or ads-only choice to Search Allow, Training Disallow AI Training and Agent Block on pages with ads. A previously unselected switch maps all three to Allow. Cloudflare is deprecating its earlier Managed Robots.txt feature in favour of Bot Preference Sync. It says it will prompt existing Managed Robots.txt customers to review and confirm their preferences before transitioning. The Bot Management API documentation now exposes ai_training: "disallow" and bot_preference_sync_enabled. Those documented fields and the announcement establish the intended product configuration; Index Us did not sign in to a customer dashboard or verify any individual domain’s migrated state.

For new domains, Cloudflare’s recommended preset depends on whether the site monetises pages using ads. Its published table offers Search Allow for both types. For ad-supported sites it offers Training Disallow AI Training and Agent Block on pages with ads; for other sites it offers Allow for those two categories. These are editable recommendations, not an assertion that every site will receive a permanent policy. An operator should inspect the resulting settings after onboarding, especially when its existing robots.txt includes custom crawler rules.

Apple and Google have documented use controls; Bing remains a gap

The mixed-use route depends on each crawler operator’s stated separation of uses. Apple’s Applebot documentation says a robots.txt rule for Applebot-Extended controls use of Applebot-crawled pages for training Apple’s foundation models. Applebot-Extended does not make separate page requests; it acts as a use signal. Apple says disallowing it does not remove pages from search results or affect Apple Search ranking. Apple’s page-level nosnippet and paywall signals govern parts of generative output separately, so a training opt-out is not a general summary opt-out.

Google’s crawler documentation similarly describes Google-Extended as a robots.txt product token rather than a separate HTTP user agent. Google says its preference governs training future Gemini models and grounding certain Gemini products from the Search index, without affecting inclusion or ranking in Google Search. That makes the practical consequence broader than model training alone for those named Gemini uses. It does not establish how every Google AI search feature will treat a site, and Cloudflare’s mention of a separate generative-search toggle should not be mistaken for the Disallow AI Training setting.

The unresolved exception is Bing. Cloudflare says Microsoft’s documented NOARCHIVE page meta tag can express a no-training preference and that Microsoft is working on domain-level robots.txt support targeted for early 2027. Until then, Cloudflare says selecting Disallow AI Training does not automatically convey a Bing no-training preference through robots.txt. This is the same practical gap its earlier Training Block had for Bingbot. Microsoft’s 2023 NOARCHIVE explanation says tagged content is excluded from Bing Chat answers and foundation-model training while ordinary Bing search can still index it. A publisher who also wants AI answer inclusion faces a separate tradeoff. Because that explanation is dated, a site deciding today should recheck current Bing controls and its page metadata before changing policy.

Check the resulting policy against the outcome you want

For a publisher, the immediate work is to define the desired result per channel before moving a setting. Traditional search discovery, model training and AI-generated summaries are different uses. A site may want search referrals while refusing training; it may also want to appear in AI summaries with citations, or refuse excerpts from a particular page. Cloudflare’s new setting does not settle all three choices at once, and it does not give the site control over material already collected.

Start with the domain’s current Search, Training and Agent selections, then compare them with Cloudflare’s migration table. Inspect the public robots.txt that Bot Preference Sync emits, preserving unrelated custom directives. Check representative ad-supported and non-ad pages for the intended network treatment; the ads-only option relies on Cloudflare’s page classification. Check the site in ordinary search before and after any change, and use the crawler operators’ documented dashboards or controls to verify their stated handling of training and generated answers. Log what was observed rather than treating an announced migration as proof that a particular account changed correctly.

There is also a decision about trust. Disallow AI Training is a suitable preference when the operator is willing to rely on a mixed-use crawler’s documented separation of uses while keeping its search access. Block is the stronger access refusal but can remove ordinary search visibility. The Bing exception needs a page-level or operator-specific decision for now. Cloudflare’s release makes that tradeoff explicit; it does not eliminate the need to test the settings, read each crawler’s actual terms and decide what exposure the site can accept.

This analysis is based on the linked public announcements and documentation. Index Us did not test Cloudflare dashboard rollout, migrate a customer account, inspect private crawler logs or verify downstream training behaviour.

Sources

1. Cloudflare — Accountable mixed-use AI crawlers announcement
2. Cloudflare — July AI traffic options
3. Cloudflare — Bot Preference Sync
4. Cloudflare Developers — Bot Management update API
5. Apple Support — Applebot controls
6. Google Developers — common crawlers and Google-Extended
7. Microsoft Bing — NOARCHIVE and AI use
8. RFC Editor — Robots Exclusion Protocol

Last reviewed: 16 Sept 2026. See our editorial policy for how we test, source and correct our work.