Cloudflare published a post on 21 August describing Bot Preference Sync, a feature that takes the AI-bot access policy an operator already set in the dashboard and writes it into the site’s robots.txt file, prepended above whatever the operator wrote by hand. The convenience framing undersells what changed: the dashboard becomes the source of truth, and robots.txt becomes a file the platform generates rather than one an operator maintains.
What Is Bot Preference Sync?
Bot Preference Sync is a Cloudflare feature, announced 21 August 2026 and rolling out to every plan within the following week, that writes an operator’s existing AI-bot dashboard settings directly into their site’s robots.txt file. It does not set the policy; it publishes whatever policy the operator already chose for Search, Agent, and Training crawlers. If a site already has a robots.txt file, Cloudflare’s additions are prepended above the existing content, so any Disallow lines an operator wrote by hand are kept in place.
The File Becomes Output
Cloudflare frames the reasoning, in the announcement post, as a practical complaint: “We recognize that it’s cumbersome to maintain multiple layers of protection on your website.” The tool doesn’t infer a policy from robots.txt; it reads the policy already in the dashboard and writes robots.txt from it, prepending its output rather than reconciling the two. Cloudflare describes it as available “to all customers from the Free tier to Enterprise.” inimino covered Cloudflare’s earlier crawler controls in June, when AI Crawl Control landed in beehiiv.
Block and Disallow Are Not the Same Control
Search and Agent each offer three settings: Allow, Block on pages that serve ads, or Block everywhere. Training adds a distinct option: Disallow, described as writing “a ‘no training’ preference … so that cooperating mixed-use crawlers who take the extra Transparency step can still access your content for search indexing.” A Block closes the door at Cloudflare’s network edge; a Disallow only writes a request into a text file, which a crawler can choose to honor.
Four Conditions for Crawlers That Do Both
A crawler running Search and Training under one user agent creates a problem: an operator who Disallows Training doesn’t want to lose Search too. A cooperating mixed-use crawler can still reach such a site, but only by meeting four conditions Cloudflare published in the 21 August post:
- “The bot must respect, via any mechanism, a ‘no training’ preference in robots.txt”
- “They give site owners a way to opt out of AI summaries”
- “They provide URL-level visibility into which pages were made available for training, as well metrics on search results”
- “They can show publicly that Disallowing Training does not hurt your traditional search results”
The fourth goes furthest: public proof that honoring a no-training preference costs nothing in search rankings — close to what publishers negotiating directly with search engines haven’t secured. Our own reporting in July found no clean way for publishers to opt out of AI use without also losing ordinary search visibility. Cloudflare’s conditions put a version of that demand in writing, as a rule for its own network, not a documented product from any search engine.
What Changes on September 15
Cloudflare’s developer changelog, published 1 July 2026, sets a default: starting 15 September 2026, bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed — and that default applies to new domains onboarding to Cloudflare, not every site already on the network. Mixed-use crawlers, in Cloudflare’s phrasing, “will be affected by the new defaults to block Training.” The 21 August post doesn’t mention 15 September; that date is from the July changelog, a separate document. The Search/Agent/Training taxonomy behind both was fixed 1 July and isn’t new this week — the new parts are the sync tool and four conditions.
Neither document names a crawler that meets the four conditions, or has been asked to. What moved this week isn’t which crawlers get blocked; it’s where the decision lives. The file that used to be the enforcement point is now a printout of a decision made elsewhere, and Cloudflare does not say who has agreed to prove anything.