A founder working on a laptop at a street cafe table
One toggle set a year ago, one deadline, and a failure mode that looks exactly like a Google penalty.

By Cody Wise, Founder, Wise Media
Last updated: August 24, 2026

If you ever clicked Block AI Bots on Cloudflare, Googlebot gets blocked with it on September 15, 2026. Cloudflare now judges multi-purpose crawlers on every behaviour they perform, and the most restrictive rule wins. Googlebot crawls for Search and for Training, so a Training block catches it. You can opt out in your zone’s Security settings before September 15. Here is exactly how.

Summary

  • Cloudflare replaced its old one-click “Block AI bots” switch with three independently controllable categories: Search, Agent and Training. Those controls are live now for every customer, including the Free tier.
  • Two separate things change on September 15, 2026, and most coverage has conflated them into one alarming claim that is not accurate.
  • Change one is narrow. For new domains onboarding to Cloudflare, Training and Agent will be blocked by default on pages that display ads, and Search will remain allowed by default.
  • Change two is the one that breaks existing sites. Multi-purpose crawlers will be judged on all of their behaviours, and the most restrictive applicable rule wins.
  • Googlebot, Applebot and BingBot are multi-purpose crawlers that combine Search with Training. Anyone who blocked Training, including through the legacy Block AI bots service, will block them.
  • The fix is an explicit opt-out inside your zone’s Security settings, confirming you want no changes to Training crawlers that also crawl for Search. Cloudflare says this can be done any time leading up to September 15.
  • This is edge-level blocking. It happens before robots.txt is ever read, so a permissive robots.txt file does not protect you.
  • Cloudflare states that more than 20% of web domains sit behind its network, which is why a default change of this kind is worth ten minutes of your attention.

Table of Contents

Does This Actually Affect You?

Only if you, or somebody who once had access to your Cloudflare account, blocked AI training. That is the whole test. If nobody ever touched the AI bot controls on your zone, Cloudflare’s published post does not say anything is going to start blocking Googlebot on your existing site.

The problem is that most business owners do not know the answer. The toggle was popular. Through 2024 and 2025 a steady stream of articles told owners to protect their content from AI scraping, and Cloudflare made it one click. Developers flipped it during launches, agencies flipped it as a hardening step, owners flipped it after reading a newsletter. Almost nobody documented it, and almost nobody has looked since.

Your situationAffected on Sept 15?What happensWhat to do
You are on Cloudflare and have never opened Security settingsNo, based on what Cloudflare has publishedCloudflare describes the new defaults as applying to new domains onboarding to Cloudflare. It does not state that untouched existing zones get new blocks.Verify rather than assume. Open the settings and read the state of Search, Agent and Training yourself.
You clicked the legacy “Block AI bots” toggleYesThat preset covered single-purpose training crawlers. From September 15 the most restrictive rule wins, so Googlebot, Applebot and BingBot get blocked too.Mark the opt-out in Security settings before September 15, or deliberately unblock Training.
You blocked Training under the new three-category controlsYesSame mechanic. A Training block now catches every crawler performing any training behaviour, including the ones powering search indexes.Decide whether the Training block is worth losing search crawling. If not, opt out.
You are adding a NEW domain to Cloudflare after September 15Yes, but narrowlyTraining and Agent will be blocked by default on pages that display ads. Search remains allowed by default.Set the three categories intentionally during onboarding rather than inheriting them.
You are not on Cloudflare at allNoThis is a Cloudflare dashboard setting. It does not exist on your stack.Confirm your nameservers. Plenty of owners are on Cloudflare without knowing.
Who is affected by the September 15, 2026 Cloudflare AI traffic default changes.

Read the middle two rows again. Those are the ones that matter, and they describe an action taken in the past by somebody who has probably forgotten they took it.

What Cloudflare Is Actually Changing on September 15 (Two Changes, and the Press Got Them Mixed Up)

A server rack with patch cables and status lights in a data centre aisle
Cloudflare classifies Googlebot as both Search and Training. Block one and you block the other.

There are two changes, not one. They have different scopes, different victims and different levels of urgency. Cloudflare set them out in its post on new AI traffic options for all customers, published July 1, 2026 by Jin-Hee Lee and Bryan Becker. The same change is logged in the Cloudflare developer changelog entry for AI traffic options.

Change one: new domains, ads pages, Training and Agent blocked by default

“On September 15, 2026, we’ll be setting new defaults for each of these three classifications. For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.”

Cloudflare, “Your site, your rules: new AI traffic options for all customers”

Three qualifiers are doing enormous work in that sentence. New domains onboarding to Cloudflare. Pages that display ads. Search remains allowed. Strip any one of them out and you get the headline that has been circulating, which is considerably scarier than the actual policy.

Cloudflare’s reasoning for tying the default to ad-bearing pages is straightforward. As the post puts it, “An ad is a signal that a website owner meant for a person to land there and see it, something monetizable that fuels the business.” An ad unit is a machine-readable declaration that a human was the intended audience. Defensible, and narrow. Most local service websites in Canada do not run display ads at all, so this first change is close to irrelevant for them.

Change two: multi-purpose crawlers judged on all their behaviours, most restrictive rule wins

This is the one that will actually knock sites out of Google, and it applies to zones that already exist.

“Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners. Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).”

Cloudflare, “Your site, your rules: new AI traffic options for all customers”

There is no ads-page qualifier here. There is no new-domain qualifier. The trigger is a choice you may have made months ago, through a toggle that used to mean something narrower than it will mean in a few weeks. Cloudflare is explicit that the legacy preset had a smaller footprint: “The managed preset to ‘Block AI bots’ that we’ve announced in the past included single-purpose bots that crawled data for model training.” Single-purpose. That is what you agreed to. From September 15 the same setting reaches further, and it reaches the crawler that feeds Google Search. Cloudflare’s own documentation on the Block AI bots configuration is where the legacy behaviour is described.

Search Engine Journal reported independently that Cloudflare’s AI crawler rules can block Googlebot, and that reporting has described live tests where Googlebot and Bingbot received an HTTP 403 response. Treat that as reporting rather than a Cloudflare statement, but it lines up exactly with what the policy text describes.

The correction: what is NOT changing

Here is the part almost every summary has got wrong, and the reason this article exists.

  • Cloudflare’s post does not say that existing customers who have changed nothing will have Training or Agent blocked. The new default applies to new domains onboarding to Cloudflare. If you have never opened those controls, the published policy does not describe a new block landing on your zone.
  • The ads-page default is not a site-wide default. It is scoped to pages that display ads.
  • Search stays allowed by default. Cloudflare is not switching off search indexing for anybody. The Googlebot problem comes entirely from a Training block spilling over onto a crawler that also does Search.
  • Paid plans are not exempt. The multi-purpose crawler rule is about what you selected, not what you pay. A Business or Enterprise zone with a Training block behaves the same way as a Free zone with a Training block.

The accurate summary is narrow and unglamorous: if you blocked AI training, you are about to block Google. If you did not, you probably are not affected, and you should still go and look, because you may not be the only person who has ever had the keys to your account.

Why Blocking AI Training Blocks Google Search

Because Googlebot is one crawler doing several jobs, and Cloudflare has stopped pretending otherwise.

Historically a bot arrived at the edge and got a single verdict against a single identity. That model held while crawlers did one thing each. It stopped holding when the same user agent began feeding a search index, an answer engine and a training pipeline at once.

Cloudflare’s answer is to classify by behaviour instead of by name. A crawler is evaluated against every category it belongs to. Googlebot belongs to Search. It also belongs to Training. Two rules apply, and the more restrictive one decides. Allow Search, block Training, and Googlebot gets refused.

Cloudflare’s recommended fix is aimed at bot operators, not at you: “If a company runs automation that builds Search indexes, acts as an Agent, and collects data to Train their models, then we strongly encourage that company to separate the automation into three separate crawlers.” Reasonable ask, and completely outside your control. You cannot make Google split Googlebot. You can only decide what your edge does when the combined crawler arrives.

One related change matters. Cloudflare has redefined its Verified label: “non-verified bots are still default blocked, but we are no longer viewing Verified as ‘default allowed.’ Now, the Verified label makes a bot allowable with its relevant category, meaning the allowed category (e.g., allowing Search) will determine what is allowed to access a website.” Verified used to be a pass. Now it is only eligibility, and the category decides. If your mental model was “Googlebot is verified, therefore Googlebot is fine,” it is out of date. The Cloudflare verified bots documentation covers how verification works.

What a 403 to Googlebot actually does to you

It does not delete your site from Google on day one. It does something slower and harder to notice, which is arguably worse.

Googlebot requests a URL and receives a 403 Forbidden. That is a failed fetch. Google cannot see current content, so it keeps serving whatever it last had. For days or weeks nothing visibly changes in the results, and you have no reason to suspect anything.

Then the failures accumulate. Crawl rate falls, because Google reduces request volume against a host that keeps refusing it. New pages never get discovered, because the crawler cannot read the pages linking to them. Updates never register. Eventually unfetchable pages drop out of the index, and traffic declines in a pattern that looks like an algorithm update, which is exactly the wrong diagnosis to reach for.

Recovery is not a switch. Google has to rediscover, recrawl and reassess pages it already deprioritised, on its own schedule, at a crawl rate it already reduced. Coming back takes materially longer than going down. That is why the deadline matters more than the severity. An hour of work before September 15 is cheap, and a recovery project afterwards is not. If organic search is part of how you get customers, the logic behind ongoing website growth and SEO support applies here in miniature. Somebody has to watch the infrastructure, not just the content.

Search, Agent and Training: What Each Category Actually Controls

Three categories, three independent switches, one of which carries a consequence the other two do not. Cloudflare’s definitions are quoted below verbatim, because paraphrasing them is how the confusion started.

CategoryCloudflare’s definitionShould a small business block this?Why
Search“any behavior that collects or indexes your content, so it can answer questions about it later. The key is that Search is proactively building a database of your site to later respond to queries with. Site owners should expect to get referral traffic or other equitable compensation as a result.”No. Keep it allowed.This is the category that produces the traffic you want. For a local or service business, blocking Search removes you from the results your customers use to find you. No upside.
Agent“automated behavior that is acting, usually in real time, on a person’s behalf, to get something done right now. This includes chat fetch bots (e.g., ChatGPT-User) and browser-use agents (e.g., Gemini or Claude driving Chrome).”Judgement call.An agent is usually a real person’s request arriving through software. If you run a booking flow, quote form or checkout an agent could complete for a customer, blocking Agent blocks the customer. Protecting proprietary content from real-time retrieval makes blocking more defensible.
Training“a crawler taking your content to train or fine-tune a model. The key is that your data is permanently absorbed into the underlying architecture of the AI to improve its capabilities.”This is the real decision, and where the danger sits.Training is the only category with a genuine argument on both sides. It is also the one that now catches Googlebot, Applebot and BingBot through the most restrictive rule. Block it without the explicit opt-out and you pay for the principle with your search visibility.
Cloudflare’s three AI traffic categories, with a practical verdict for small and mid-sized business websites.

For most Canadian service businesses a Training block buys very little. Your service pages, about page and location pages are not the corpus anybody is fighting over. Publishers with large archives of original reporting have a real interest in that fight. A plumbing company in Calgary does not, and the cost of getting the setting wrong is entirely one-sided.

Cloudflare has also added a content-use signal to its managed robots.txt, which expresses intent rather than enforcing it. The managed block now reads:

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

The three content-use levels are immediate, meaning interact, store and reuse nothing; reference, the default, meaning index, excerpt, and link back; and full, meaning summarize and reproduce. Useful to know it exists. It is a declaration of preference in a text file, and it is a separate layer from the enforcement happening at the edge.

How to Check and Fix Your Settings Before September 15

Ten minutes, eight steps. One caveat first: Cloudflare’s post does not publish a click-by-click menu path and does not name the specific toggle in the new interface. I am not going to invent a label that might not exist. What Cloudflare does give is a deep link into the right screen, plus the names of the three categories to look for once you are there.

  1. Confirm you are actually on Cloudflare. Look up your domain’s nameservers with a public DNS tool or a WHOIS query. If they resolve to Cloudflare nameservers, this applies to you. Plenty of owners are on Cloudflare because a previous developer put them there and never mentioned it.
  2. Open your zone’s Security settings. Use the deep link Cloudflare provides: https://dash.cloudflare.com/?to=/:account/:zone/security/settings. If you cannot log in, stop here and find who can, because that is the actual blocker.
  3. Find the AI traffic controls and read the current state. Look for the Search, Agent and Training categories. Write down what each is set to before changing anything. If a legacy Block AI bots setting is still enabled, that is your answer.
  4. Decide per category using the table above. Search allowed for almost every business. Agent decided on whether real-time agent traffic could plausibly be a customer. Training decided deliberately, knowing a block reaches Googlebot from September 15.
  5. Mark the opt-out explicitly if you want no change. Cloudflare’s wording: “if a website owner wants to opt out of these new default configurations, they can easily mark this in their Security settings any time leading up to September 15, which will confirm that they want no changes on Training crawlers that also crawl for Search purposes.” Cloudflare adds that it will “continue to notify customers of the upcoming change to defaults as we approach September 15.” Do not wait for the notification.
  6. Check your managed robots.txt content signal. If Cloudflare manages your robots.txt, confirm the Content-Signal line reflects your intent. It is a separate layer from edge enforcement, and the two disagreeing is a mess to debug later.
  7. Verify Googlebot is getting through. In Google Search Console, run URL Inspection on a live URL and use the live test. If the fetch fails or returns a forbidden response, your edge is refusing Google right now.
  8. Recheck after September 15. Put a reminder in your calendar for September 16. Run the live test again, then check Crawl Stats a week later. Defaults have a way of quietly reapplying during migrations and account transfers.

If step two stops you because nobody at your company can log in to Cloudflare, that is a bigger finding than the setting itself. Access to your own infrastructure is not a technical detail. It is ownership, and it is the first thing we audit on any website build or rebuild engagement.

How to Confirm Googlebot Is Not Being Blocked

Verify from Google’s side, not Cloudflare’s. A settings screen tells you what you configured. It does not tell you what Googlebot received at the edge, and the gap between those two things is where sites disappear.

Search Console URL Inspection, live test. The fastest and most direct signal available to you. Inspect a live URL, run the live test, and read the crawl response Google reports back. A successful fetch with a rendered page means Google reached your origin. A fetch failure, a forbidden response, or a page rendering blank means something in front of your server refused the request. Test more than the homepage. Test a service page, a location page and a blog post, because rules can apply unevenly across paths.

The Crawl Stats report. Found under Settings in Search Console, this is your trend view. Watch total crawl requests, the breakdown of response codes, and host status. A rising share of 4xx responses, or a crawl request line that steps down and stays down, is the signature of an edge-level block. Crawl Stats is retrospective, so it confirms a problem rather than catching it on day one. That is why the live test comes first.

Reverse DNS verification. If you are reading server logs and want to know whether a request claiming to be Googlebot really was, follow Google’s documented method for verifying Googlebot and other Google crawlers. Reverse lookup the IP, confirm it resolves to a Google domain, then forward lookup that hostname and confirm it returns the original IP. User agent strings are trivially spoofed. The DNS round trip is not.

Server and edge logs. Ground truth. Filter for Googlebot user agents and look at the status codes returned. A clean site returns mostly 200 and 304 responses. A wall of 403s is not ambiguous. Check your edge logs too, because a request refused at the edge may never appear in origin server logs at all. That is the trap: hosting logs can look perfectly healthy while Google is turned away one hop upstream.

A Note for Canadian Small Businesses

For a local service business in Calgary, Edmonton, Red Deer or anywhere else in Alberta, blocking Search is close to self-harm and blocking Training buys you almost nothing.

Think about what is on the site. Service descriptions. A service area list. A team page. Reviews. Contact details. That content exists to be found by somebody in your city with a problem you solve, at the moment they go looking. Every one of those pages is worth more indexed than protected. The publishers driving the AI content debate are defending large archives of original material with genuine licensing value. Real argument, not your argument.

The bigger issue in the Canadian SMB market is not philosophy. It is that a large share of these websites were built by somebody no longer involved. A freelancer who moved on. An agency the business left three years ago. A relative who “knows computers.” Any one of them could have flipped Block AI bots during setup as a sensible-looking precaution and never mentioned it, because at the time it genuinely was one. You cannot audit a decision you were never told about.

So use this deadline for something more useful than one toggle. Run an ownership and access audit across the four accounts that control whether your business exists online:

  • Domain registrar. Registered in your company name, with an email address you control, and not expiring quietly next year.
  • DNS and Cloudflare. You hold an account with admin rights. Not your developer. You.
  • Hosting. You can log in, you know what plan you are on, and you know where the backups are.
  • Google Search Console and Google Business Profile. You are a verified owner, not a delegated user on somebody else’s account.

This is the same checklist that runs at the start of every agency switch we handle. Half the friction in moving providers has nothing to do with design or code and everything to do with who holds the credentials. If you are considering a change, or you inherited a site nobody fully understands, a structured design and build engagement should begin with taking ownership of the stack before a pixel moves. A website is not a finished object. It is infrastructure sitting on platforms that change their defaults, sometimes with six weeks of notice and a blog post.

The September 15 Cloudflare Checklist

  1. Confirm whether your domain resolves through Cloudflare nameservers.
  2. Confirm you personally have admin access to the Cloudflare account. If not, get it.
  3. Open your zone’s Security settings using Cloudflare’s deep link.
  4. Record the current state of Search, Agent and Training before changing anything.
  5. Check specifically for a legacy Block AI bots setting that is still enabled.
  6. Keep Search allowed unless you have a deliberate, documented reason not to.
  7. Make a conscious decision on Agent based on whether your site has a booking, quote or checkout flow.
  8. Make a conscious decision on Training, knowing a block now reaches Googlebot, Applebot and BingBot.
  9. Mark the explicit opt-out if you want no change to Training crawlers that also crawl for Search.
  10. Confirm your managed robots.txt Content-Signal line matches your intent.
  11. Run a Search Console URL Inspection live test on at least three different page types today.
  12. Calendar a recheck for September 16, then review Crawl Stats one week later.
  13. Write down what you set and why, so the next person to touch this account is not guessing.

Common Mistakes

Assuming a paid plan makes you safe. It does not. The multi-purpose crawler rule is triggered by what was selected on the zone, not by what the zone costs. A Business plan with a Training block and a Free plan with a Training block behave identically.

Assuming robots.txt is the same thing as the Cloudflare setting. The most expensive misunderstanding here. Robots.txt is a file on your server that well-behaved crawlers voluntarily read and obey. The Cloudflare control is enforcement at the network edge. A blocked request is refused before it reaches your origin, which means before robots.txt is ever fetched. A permissive robots.txt gives you no protection against an edge block. Different layer, different mechanism, different failure mode.

Assuming you would notice. You would not. Your site stays up, pages load normally for humans, no error appears in a dashboard, no email arrives. The failure is visible only in crawl data, a report most owners have never opened. By the time the traffic decline is obvious enough to investigate, weeks have gone by.

Blocking Training to “protect content” without pricing the trade. Protecting content is a legitimate position. It is not a free one. If blocking Training also removes you from the index that sends you customers, you have paid for a principle with your pipeline. Make that trade deliberately or not at all.

Treating this as the same issue as robots.txt AI crawler directives. That wider conversation is a separate topic with separate mechanics. This is one dashboard setting inside Cloudflare and one date. Conflating them is how people confidently fix the wrong layer and stay blocked.

Doing it on September 14. Cloudflare says the opt-out can be marked any time leading up to September 15. Deadlines create support queues. Do it this week, verify, then recheck after the date.

Frequently Asked Questions

Will Cloudflare block Googlebot on September 15?

Only for customers who selected to block Training, including through the legacy Block AI bots service. Cloudflare states that from September 15, 2026, multi-purpose crawlers such as Googlebot, Applebot and BingBot will be blocked for those customers, because the defaults are enforced by the most restrictive applicable rules. If you never blocked Training, this does not describe your zone.

Does this affect me if I am on Cloudflare’s free plan?

Plan tier is not the deciding factor. The new three-category controls are available to all customers including the Free tier, and the multi-purpose crawler rule applies based on whether Training is blocked on your zone, not on what you pay. A free zone with no Training block is in the same position as a paid zone with no Training block.

I never changed any Cloudflare settings. Am I safe?

Based on what Cloudflare has published, yes. The new defaults are described as applying to new domains onboarding to Cloudflare, on pages that display ads, with Search remaining allowed. Cloudflare’s post does not say untouched existing zones will get new Training or Agent blocks. Still verify, because a previous developer may have changed settings you never knew about.

How do I opt out before September 15?

Open the AI traffic controls in your zone’s Security settings using Cloudflare’s deep link, then mark the opt-out confirming you want no changes to Training crawlers that also crawl for Search purposes. Cloudflare says this can be done any time leading up to September 15. Cloudflare has not published a click-by-click menu path, so look for the Search, Agent and Training categories.

Is blocking AI training bad for SEO?

From September 15, 2026, on Cloudflare, yes. Blocking Training triggers the most restrictive applicable rule against multi-purpose crawlers, which means Googlebot, Applebot and BingBot get blocked with the training-only crawlers. That removes search engine access to your site. For a local or service business, the cost of losing search crawling almost always exceeds the benefit of blocking training.

What is the difference between Search, Agent and Training?

Search is behaviour that collects or indexes your content to answer questions about it later, and it should generate referral traffic. Agent is automated behaviour acting in real time on a person’s behalf, including chat fetch bots and browser-use agents. Training is a crawler taking your content to train or fine-tune a model, permanently absorbing it into the model architecture.

Does robots.txt stop this?

No. Cloudflare’s blocking happens at the network edge, before a request reaches your server and therefore before robots.txt is read or considered. Robots.txt is a voluntary instruction file for crawlers that choose to obey it. Editing robots.txt cannot undo an edge-level block. The fix has to happen in your Cloudflare Security settings, not in a text file.

How do I know if Googlebot is being blocked right now?

Run a live test in Google Search Console URL Inspection on several page types and read the crawl response. A forbidden response or a failed fetch means something is refusing Google before your server answers. Confirm the pattern in the Crawl Stats report by checking host status and the share of 4xx responses, then check your edge logs.

What happens to my rankings if Googlebot gets 403 errors?

Nothing visible at first, then a slow decline. Google keeps serving previously indexed content while fetches fail, reduces crawl rate against a host that keeps refusing it, stops discovering new pages, and eventually drops pages it cannot fetch. Recovery is not instant, because rediscovery and recrawling happen on Google’s schedule at a reduced crawl rate.

Do I need to do anything if my site is not on Cloudflare?

No. This is a Cloudflare dashboard setting and it does not exist elsewhere in your stack. Confirm it properly rather than assuming, by checking your domain’s nameservers. Cloudflare states that more than 20% of web domains sit behind its network, and many owners are on it without knowing because a previous developer configured it.

Conclusion

The scary version of this story is wrong. Cloudflare is not switching off search indexing for the internet on September 15. Search remains allowed by default, and the new ads-page default is scoped to new domains onboarding to the platform.

The true version is smaller and, for a specific group of owners, considerably worse. If you blocked Training at any point, including with one click on a toggle that meant something narrower at the time, Googlebot, Applebot and BingBot get blocked with it. Nothing will look broken. The only symptom is search traffic that quietly stops arriving.

The work is ten minutes. Confirm you are on Cloudflare, open your zone’s Security settings, read the state of Search, Agent and Training, mark the opt-out if you want no change, then verify with a Search Console live test rather than trusting the settings screen. Check again after September 15.

The wider lesson is worth keeping. Your website runs on platforms that change their defaults, and a decision made in five seconds two years ago can be re-scoped by somebody else’s product update. Infrastructure needs an owner who checks.

Get Your Configuration Audited Before September 15

If you do not know what your Cloudflare settings say, or you cannot log in to find out, that is the finding. Wise Media runs technical and configuration audits for Canadian businesses covering DNS, CDN and edge settings, crawlability, Search Console access, and who actually owns each account in your stack. We tell you what is set, what it will do on September 15, and what to change.

Start with the Wise Media intake form and tell us your domain. We will check whether you are behind Cloudflare, whether Googlebot is currently being served successfully, and whether anything in your configuration is set to break in September. If the audit turns into a rebuild or an ongoing engagement, our website growth packages cover the monitoring so a platform default change never becomes a traffic problem you find out about in October.