By Cody Wise, Founder, Wise Media. Published 9 September 2026. Wise Media builds and maintains websites for Canadian founders, agencies, trades and short-term rental operators.

Allow the search crawlers, decide deliberately about the training crawlers, and never block a bot you have not looked up first. In 2026 the major AI companies run separate user agents for training, for search, and for user-initiated fetches. Blocking the wrong token removes your business from ChatGPT answers while doing nothing at all about training. Here is every token that matters and what blocking each one actually costs you.

Summary

A data centre cold aisle lined with server racks and blue status lights, representing AI crawler traffic hitting a website
Every line you write in robots.txt is a request sent to machines like these. It is a request, not a lock.
  • The crawlers are split by job. OpenAI runs GPTBot for training, OAI-SearchBot for ChatGPT search, ChatGPT-User for on-demand fetches and OAI-AdsBot for ad review. Anthropic splits the same way with ClaudeBot, Claude-SearchBot and Claude-User.
  • Google-Extended is not a crawler. It is a control token, and Google states plainly that it does not affect inclusion in Google Search. Blocking it does not remove you from AI Overviews.
  • The AI Overviews opt-out moved. Since 31 August 2026 it lives in Search Console as the Search generative AI control, available to every site worldwide. It is not a robots.txt job.
  • For most Canadian service businesses the correct answer is to allow everything. You want to be the answer, not the excluded page.
  • robots.txt is voluntary. OpenAI and Perplexity both state in their own documentation that their user-initiated fetchers may ignore it.
  • Check Cloudflare before you touch robots.txt. A default change on 15 September 2026 can block Googlebot, Applebot and bingbot for anyone who ever ticked the legacy Block AI Bots switch.

Table of contents

What robots.txt actually controls, and what it does not

robots.txt is a plain text file at the root of your domain that tells automated visitors which paths they may request. It controls crawling. It does not control indexing, it does not control what a model already learned two years ago, and it does not control what other websites have written about you.

That distinction matters more in the AI era than it did in the search era, because AI answers are assembled from many sources. Roughly nine in ten AI-generated answers cite third-party pages rather than the brand’s own site. Blocking a crawler from your domain does nothing about the directory listing, the review site or the Reddit thread that mentions you.

Three things robots.txt cannot do

  1. It cannot remove a page from an index. That is what a noindex directive is for. A disallowed page can still appear as a bare URL if enough other pages link to it.
  2. It cannot retroactively unlearn anything. Anthropic’s own guidance is explicit that blocking ClaudeBot signals that future material should be excluded from training datasets.
  3. It cannot enforce anything. It is a published request. Enforcement happens at the network layer, in your CDN or firewall, not in a text file.

One more thing worth knowing before you edit it: your robots.txt is public. Anyone, including a competitor, can read yours at yourdomain.ca/robots.txt. Never use it to hide a staging site, a client portal or a pricing page you would rather nobody saw. Use authentication for that.

The four jobs an AI crawler does

Most guidance published before 2025 treats “AI bots” as one category. That framing is now the single most expensive mistake in this whole area, because the same company runs different bots for different purposes and you almost certainly want to say yes to some and no to others.

JobWhat it doesWhat you lose by blocking it
TrainingCollects pages that may be used to train a foundation modelNothing immediate. Possibly long-term brand familiarity in future models
Search indexingBuilds the live index the assistant searches when answering a questionYour business stops appearing in that assistant’s cited answers
User-initiated fetchRetrieves a page because a human pasted or asked about that URLA prospect who pastes your URL gets “I cannot access that page”
Ads verificationChecks the safety of a landing page submitted as an adYour ad landing page fails review

Read that third row twice. A prospect who copies your quote page into ChatGPT and asks “is this a fair price” is the highest-intent traffic you will ever get. If your robots.txt blocks the fetcher, that conversation ends with an error message.

The 2026 AI crawler reference table

Every token below is taken from the vendor’s own published documentation as of 9 September 2026, not from a third-party list. Tokens change, so verify against the source links at the end of this article before you paste anything into production.

CompanyTokenJobWhat blocking it means
OpenAIGPTBotTrainingContent excluded from future foundation model training. No effect on ChatGPT search
OpenAIOAI-SearchBotSearchYou will not be shown in ChatGPT search answers, though you can still appear as a navigational link
OpenAIChatGPT-UserUser fetchOpenAI states robots.txt rules may not apply here, because a human asked
OpenAIOAI-AdsBotAds reviewPages submitted as ChatGPT ads fail safety validation. New for 2026
AnthropicClaudeBotTrainingFuture material excluded from training datasets
AnthropicClaude-SearchBotSearchYour content is not indexed for Claude’s search results
AnthropicClaude-UserUser fetchClaude cannot retrieve your page in response to a user query
GoogleGooglebotSearch, and AI Overviews groundingYou leave Google Search entirely. Do not block this
GoogleGoogle-ExtendedControl token onlyExcludes you from Gemini training and grounding. Explicitly no effect on Google Search or AI Overviews
GoogleGoogle-CloudVertexBotOwner-requestedOnly crawls sites at their own owner’s request for Vertex AI agents. Not an ambient scraper
PerplexityPerplexityBotSearchYou are not surfaced or linked in Perplexity results. Not used for model training
PerplexityPerplexity-UserUser fetchPerplexity states this fetcher generally ignores robots.txt rules
MicrosoftbingbotSearch and CopilotThere is no separate Bing AI token. Microsoft’s published route is the NOARCHIVE and NOCACHE meta directives instead
AppleApplebotSearch and trainingYou leave Siri, Spotlight and Safari search
AppleApplebot-ExtendedControl token onlyExcludes you from Apple foundation model training. Pages remain in search results
Metameta-externalagentTraining and indexingExcluded from Meta AI training
Metameta-externalfetcherUser fetchMeta AI cannot retrieve a URL a user supplied
AmazonAmazonbotTraining and AlexaExcluded from Amazon AI products
ByteDanceBytespiderTrainingExcluded from ByteDance model training
Common CrawlCCBotOpen datasetExcluded from the public corpus many smaller models train on
MistralMistralAI-UserUser fetchMistral cannot retrieve a user-supplied URL
DuckDuckGoDuckAssistBotAssistantExcluded from DuckDuckGo’s assistant answers
Cody Wise working at a multi-monitor founder workstation reviewing site configuration
The whole audit is a fifteen minute job. It is skipped because nobody owns it, not because it is hard.

Three sentences worth reading in the vendors’ own words

  • OpenAI, on OAI-SearchBot: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”
  • Google, on Google-Extended: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”
  • Perplexity, on Perplexity-User: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”

That third quote is the most honest sentence any AI vendor has published about robots.txt. It is worth building your expectations around.

Three expensive mistakes Canadian businesses are making right now

A brushed steel office lobby turnstile with one gate arm open and one closed
One gate open, one closed. The problem is that most owners cannot tell you which of theirs is which.

Mistake 1: blocking OAI-SearchBot to stop AI training

This is the most common one, and it is precisely backwards. OAI-SearchBot has nothing to do with training. It is the crawler that builds the index ChatGPT searches when a user asks a question. Blocking it removes you from ChatGPT search answers and leaves GPTBot, the actual training crawler, completely untouched.

If your goal is to stay out of training data, the line you want is User-agent: GPTBot followed by Disallow: /. Nothing else.

Mistake 2: blocking Google-Extended to get out of AI Overviews

AI Overviews and AI Mode are grounded in the regular Google Search index, fetched by ordinary Googlebot. Google’s own AI optimization guide states that a page must be indexed and eligible to show with a snippet in order to be eligible for generative features. Google-Extended governs Gemini app training and grounding, and Google says outright that it has no effect on Search inclusion or ranking.

The correct lever is new. On 31 August 2026 Google finished rolling out the Search generative AI control in Search Console to all websites worldwide. It excludes you from AI Overviews, AI Mode and generative Discover features without removing you from Search. Google’s own help documentation adds that this control does not affect training, and points you back to Google-Extended for that.

For a local service business we would almost never recommend using it. If you are troubleshooting what your visibility currently looks like across both engines, our guide to what Google’s AI search report will and will not tell you covers the measurement side.

Mistake 3: ticking a switch in Cloudflare and forgetting about it

This is the one with a deadline on it, and it is six days from the date of this article. Cloudflare’s own post on the change reads, verbatim:

Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors… Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).

Cloudflare, “Your site, your rules: new AI traffic options for all customers”, 1 July 2026

Cloudflare classifies Googlebot, Applebot and bingbot as multi-purpose crawlers because they serve both search and training. Under most-restrictive matching, a customer who blocked Training blocks all three. If you flipped that switch a year ago because a blog post told you to protect your content, you have until 15 September to say otherwise in your Cloudflare Security settings. We wrote the full walkthrough here: if you ever clicked Block AI Bots on Cloudflare, September 15 will deindex you from Google.

The general lesson: robots.txt is not the only place a crawler gets blocked. Your CDN, your firewall and your WordPress security plugin all have their own bot rules, and they override the polite text file every time.

So which crawlers should you actually allow?

Macro photograph of a network patch panel with some ports lit and connected and others dark and empty
Some ports lit, some dark. A good robots.txt is a deliberate pattern, not a default nobody has read.

The honest answer depends on whether your content is your marketing or your product. Three profiles cover almost every Canadian business we work with.

Profile A: allow everything (most businesses)

If you are a trade, an agency, a realtor, a clinic, a short-term rental operator, a SaaS company or any business whose website exists to win work, allow everything. Your pages are marketing collateral. Being quoted by an assistant is the outcome you want, and there is no revenue attached to a model not having seen your services page.

# Allow all crawlers, including AI
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.ca/sitemap_index.xml

That is genuinely it. You do not need to name each AI crawler to allow it. The absence of a rule is permission. Naming them explicitly only helps if you want a documented record of a deliberate decision, which for a client site is often worth the four extra lines.

Profile B: allow search and user fetch, block training only

This is the middle position for a business with genuinely proprietary written material, an original methodology, or a large body of research it does not want absorbed into a model wholesale. You stay fully visible in AI search results while opting out of training corpora.

# Opt out of model training, stay visible in AI search
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Amazonbot
Disallow: /

# Search and user-initiated fetchers stay allowed by omission:
# OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
# PerplexityBot, Perplexity-User, Googlebot, Applebot, bingbot

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.ca/sitemap_index.xml

Note what is not in that file. Googlebot is untouched. bingbot is untouched. OAI-SearchBot is untouched. Every crawler that decides whether you appear in an answer keeps working.

Profile C: publishers and paywalled content

If people pay to read your words, robots.txt is the wrong tool and always has been. Use Profile B as a baseline, then enforce at the network layer with your CDN’s bot controls, and treat licensing as a commercial conversation rather than a technical one. Cloudflare and others now run marketplaces for exactly this. A text file will not defend a business model.

One rule that applies to all three

Never block by IP address as an opt-out. Anthropic’s documentation says this outright, and the logic applies to every vendor: if you block the IP range, the crawler cannot read your robots.txt, so it never sees the preference you were trying to express. Use the user agent token.

How to audit your own site in fifteen minutes

  1. Read your own robots.txt. Open yourdomain.ca/robots.txt in a browser. Write down every user agent named. If you see a token you cannot explain, that is your first finding.
  2. Open Cloudflare Security settings if you use Cloudflare, and check whether Training is blocked. Do this before 15 September 2026.
  3. Check your WordPress security plugin. Wordfence, Imunify, Solid Security and most hosting-level bot protection maintain their own blocklists, and several ship AI crawlers blocked by default.
  4. Check Search Console for the Search generative AI control, and confirm it is set to include unless you deliberately chose otherwise.
  5. Check whether your CDN generates a managed robots.txt that overrides or appends to yours. Cloudflare’s managed file now emits content signals of its own.
  6. Re-check in 24 hours. OpenAI states it can take roughly a day for its systems to reflect a robots.txt change. Google and Apple are similar. Do not judge a fix on the same afternoon.

If step one turns up a rule nobody at your company wrote, it almost certainly came from a plugin, a host or a previous developer. Our WordPress maintenance plan guide covers who should own this file on an ongoing basis, which is the actual reason it drifts.

Does robots.txt actually work?

Cody Wise working on a laptop by an office window in the evening with the city behind
Compliance data is contested. The vendors’ own documentation is not, and it says more than the studies do.

Partly. Here is the version we are confident stating.

The major vendors publish IP ranges so you can verify a crawler is genuinely theirs, and the large search-purpose crawlers generally do honour the file. But two of them tell you in their own documentation that their user-initiated fetchers may ignore it. OpenAI writes that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. Perplexity writes that Perplexity-User generally ignores robots.txt rules. Neither of those is an accusation from a critic. Both are the vendor’s own published position.

Beyond that, the compliance statistics circulating in 2026 come mostly from companies selling enforcement products, and they disagree with each other. We are not going to quote a number we cannot trace to a methodology. Cloudflare’s own framing in July 2026 is the useful one: network-level attribution and enforcement is “far more effective than voluntary standards like robots.txt”. That is a vendor with a product to sell saying it, and it also happens to be true.

What Cloudflare did publish with real numbers, in its July 2026 report, is worth sitting with: more than a third of crawler activity on its network still comes from mixed-use bots that do not declare their intent, and some of the most heavily crawled content categories have seen human traffic decline by as much as 40 percent in under a year.

Practical conclusion. Treat robots.txt as a stated preference that most well-behaved crawlers respect. If you need a guarantee, you need a firewall rule, not a text file.

What about llms.txt?

Short answer: it is harmless, and it is not a lever. Google added a clarification to its AI optimization guidance in June 2026 that could not be much blunter:

You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.

Google Search Central, AI features and your website

Neither OpenAI nor Anthropic has committed to reading anyone else’s llms.txt either. There is a small irony available here: OpenAI and Perplexity both publish llms.txt files for their own developer documentation, while making no claim that their crawlers consume yours. Publishing one is not the same as reading one.

Our position: if you already have one, leave it. It costs nothing. Do not spend a budget line on it, and be sceptical of anyone selling llms.txt as an AI visibility service.

What blocking actually costs

There is one piece of real research on this rather than vendor marketing. Researchers at Rutgers and Wharton studied publishers who blocked LLM crawlers and found roughly a 7 percent decline in weekly visits within the six weeks after blocking, measured independently across two datasets. Around three quarters of top news publishers had blocked LLM crawlers by the study window.

Two caveats we would want stated if we were reading this on someone else’s site. First, the study covers publishers, whose economics are not a roofing company’s economics. Second, the blocking window predates AI Overviews, so it is evidence that blocking costs traffic, not a measurement of what it costs in 2026.

The proposed mechanism is the interesting part: the dominant effect was not lost referral clicks, it was lost brand exposure inside the answers themselves. That maps directly onto what we see with Canadian service businesses. Being mentioned is worth more than being clicked, because the mention happens at the moment of decision.

Frequently asked questions

Should a small business block AI crawlers?

Almost never. If your website exists to generate enquiries, you want assistants to be able to read and cite it. Blocking makes sense when your written content is the product you sell, not the marketing for it.

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot only affects training. ChatGPT’s search results come from OAI-SearchBot, and on-demand page fetches come from ChatGPT-User. You can block GPTBot and remain fully visible in ChatGPT answers.

Does blocking Google-Extended remove me from AI Overviews?

No. Google states that Google-Extended does not impact inclusion in Google Search, and AI Overviews are grounded in the regular Search index. The lever that does work is the Search generative AI control in Search Console, which rolled out to all sites worldwide on 31 August 2026.

Do AI crawlers actually obey robots.txt?

The search-purpose crawlers generally do. User-initiated fetchers often do not, and two vendors say so in their own documentation. OpenAI states that robots.txt rules may not apply to ChatGPT-User, and Perplexity states that Perplexity-User generally ignores robots.txt rules. For enforcement rather than preference, use your CDN or firewall.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls content that may be used to train OpenAI’s foundation models. OAI-SearchBot surfaces websites in ChatGPT’s search features. They are independent settings, and blocking one has no effect on the other.

How long does a robots.txt change take to apply?

Roughly 24 hours for OpenAI by its own published guidance. Google typically recrawls robots.txt within a day for an active site. Assume one to two days across the board and do not judge the result the same afternoon.

Is there a separate Bing AI crawler I should worry about?

No. Microsoft uses bingbot for both search and Copilot. Its published route for controlling AI use of your content is the NOARCHIVE and NOCACHE meta directives rather than a distinct user agent, so blocking bingbot in robots.txt costs you Bing search with no separate AI benefit.

Do I need an llms.txt file?

No. Google has stated that Google Search, including its generative AI capabilities, does not use them. No major AI vendor has committed to reading them. Keeping one costs nothing and gains nothing measurable.

The short version

Most Canadian businesses reading this should allow every AI crawler, confirm that no plugin or CDN has quietly blocked one on their behalf, and spend the time they would have spent on robots.txt on being worth citing instead. The businesses that get quoted by assistants are the ones with clear pricing, real answers to real questions, and pages that load. None of that lives in a text file.

If you want the audit done properly, including the Cloudflare check before 15 September, our Website Growth packages cover technical SEO and AI visibility as an ongoing engagement, and our Website packages ship with this configured correctly from day one.

Tell us what your site is doing right now and we will tell you what it costs you. Start with the Wise Media intake form.

Sources

Crawler tokens and vendor policies change frequently. Every fact in this article was verified against the primary sources above on 9 September 2026. Verify before you deploy.