Free Tool
Robots.txt And Sitemap Generator
Build a robots.txt with platform defaults, your sitemap and a clear choice for every major AI crawler, plus a sitemap.xml from a list of URLs. The mistakes that quietly deindex sites are caught before you upload anything.
Site
AI Crawlers
Sitemap.xml Builder
Free, no signup. Runs in your browser, nothing you type leaves this page.
Every result has a shareable link.
What Robots.txt And Sitemap.xml Do
robots.txt is a plain text file at the root of a host, at /robots.txt, that tells crawlers which paths they may fetch. It is defined by the Robots Exclusion Protocol, published as RFC 9309 in 2022. It is a request, not access control: well behaved crawlers follow it, and anyone can still open the URLs.
sitemap.xml is the opposite list: the URLs you want crawled, in an XML format defined at sitemaps.org. robots.txt points to it with a Sitemap: line so every crawler can find it without being told.
Get either wrong and the damage is quiet. A stray Disallow: / left over from staging can take a site out of search within days, and nothing on the site itself looks broken.
How The Generator Builds The File
- Platform defaults. WordPress gets
Disallow: /wp-admin/withAllow: /wp-admin/admin-ajax.php, which matches what WordPress serves by default and keeps front end features that call admin-ajax working. Shopify gets its standard checkout, cart and account paths. Webflow and custom start clean. - The
User-agent: *group holds your Disallow and Allow rules for every crawler. - AI crawler groups. Blocked AI crawlers are listed together under a group that ends in
Disallow: /. Allowed ones get no group, on purpose: under RFC 9309 a crawler obeys only the most specific group that names it, so a named group withAllow: /would cancel your * rules for that bot. - Sitemap lines go at the end, as absolute URLs.
The Rule Matching Test
Every page you list under "pages that must stay crawlable", and every URL in the sitemap builder, is tested against your rules the way Google evaluates them: the longest matching path wins, * matches anything, $ anchors the end, and when an Allow and a Disallow match with equal length, Allow wins. A money page caught by a broad rule shows up in red.
Training Crawlers Versus Search Crawlers
AI companies run different crawlers for different jobs, and the choice is not all or nothing.
- Training crawlers collect pages to train models: GPTBot, ClaudeBot, CCBot and Bytespider. Google-Extended and Applebot-Extended are not separate crawlers. They are tokens that tell Google and Apple whether pages their normal crawlers fetch may be used for AI training.
- Search and answer crawlers fetch pages to show and cite in answers: OAI-SearchBot for ChatGPT search, ChatGPT-User when a person asks ChatGPT to open a page, Claude-SearchBot for Claude, and PerplexityBot for Perplexity.
A business that wants to be recommended by assistants usually allows the search crawlers, whatever it decides about training. Blocking Google-Extended does not remove you from Google Search or AI Overviews, which run on Googlebot.
The Mistakes It Blocks
Disallow: /on a live site. Left out unless you tick the staging box.noindexin robots.txt. Google stopped supporting it in September 2019. The line is removed and you are pointed at the meta robots tag instead.- Relative sitemap URLs. Converted to absolute when your website is set, removed when it is not.
- Blocking CSS and JavaScript. Flagged, because Google renders pages and needs those files.
- No sitemap line. Flagged with the usual sitemap URL for your platform.
- Sitemaps listing blocked or foreign URLs. Removed, because a sitemap should only contain crawlable pages on its own host.
Shipping It
Upload robots.txt to the root of each host or subdomain. On WordPress, a real file replaces the virtual one, or use your SEO plugin's editor. On Shopify, edit robots.txt.liquid and keep the defaults. On Webflow, paste it into the SEO settings. Then open Search Console, check the robots.txt report, submit the sitemap, and run URL Inspection on a key page to confirm it is crawlable.
One more limit worth knowing: blocking a URL in robots.txt does not remove it from the index. If other sites link to it, Google can still list the bare URL. To keep a page out of results, allow crawling and use noindex.
The Agency Angle
Every site launch and migration should include a robots.txt and sitemap review, and AI crawler policy is now a question clients ask. Wise Media uses this logic on client launches: platform defaults, an explicit AI policy agreed with the client, and a crawl test of the money pages before go live.
Fix What This Measured
SEO & AI Search Agency Course
The SEO & AI Search Agency Course covers the crawl layer properly: robots rules per platform, sitemap strategy for large sites, AI crawler policy you can explain to a client, and the Search Console checks that confirm it all worked.
Rather Not DIY It?
Let Wise Media Build It.
The tool shows the gap. The studio closes it: brand, site, app and AI, then the ads, email, content and search that turn them into revenue.
Website Development
Custom sites with one job per page, built fast for phones, search and AI answers.
From $3,000CAD one timeSee Website DevelopmentApp Development
Web apps, portals and dashboards on Next.js, Supabase and Stripe, installable on any phone.
From $5,000CAD one timeSee App DevelopmentPaid Advertising
Meta, Google and TikTok campaigns built on the offer and page first, reported as one number.
From $750CAD per monthSee Paid AdvertisingSEO & AI Search
Rank in Google, get named by ChatGPT and Gemini, and convert the visit after the click.
From $350CAD per monthSee SEO & AI SearchBrand Identity
Positioning, logo suite, colour, type and the guide that keeps it consistent.
From $750CAD one timeSee Brand IdentityAI Agents & Automation
Custom AI agents, LLM features and n8n automation connected to the tools you already use.
From $1,000CAD one timeSee AI Agents & AutomationContent Production
Short form video, carousels and AI clone content produced on a weekly calendar.
From $750CAD per monthSee Content ProductionEmail Marketing
Campaigns written and sent, lifecycle flows and SMS, every send tied to revenue.
From $350CAD per monthSee Email MarketingWebsite Care
Updates, daily backups, monitoring, edits and search maintenance handled every month after launch.
From $100CAD per monthSee Website CareCyber Security
Email, accounts, staff devices and company data protected, with the documents your insurer asks for.
From $250CAD per monthSee Cyber SecurityRobots.txt And Sitemap Generator Questions
Straight answers on using it and acting on the result.
Quick Answer
How Do I Block AI Crawlers With Robots.txt?
Add a group naming each AI crawler, such as User-agent: GPTBot, followed by Disallow: /. Block training crawlers like GPTBot, ClaudeBot and CCBot if you do not want your content used for training, and keep search crawlers like OAI-SearchBot and PerplexityBot allowed if you want to appear in AI answers.
- Standard. Robots Exclusion Protocol, RFC 9309
- Training tokens. GPTBot, ClaudeBot, Google-Extended, CCBot
- Search crawlers. OAI-SearchBot, Claude-SearchBot, PerplexityBot
FAQ
Frequently Asked Questions
For most sites: User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, and a Sitemap line with your full sitemap URL. Do not block /wp-includes/, themes or plugins, because Google needs those CSS and JavaScript files to render your pages.
Not from ChatGPT search. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and page visits a user asks for use ChatGPT-User. You can block GPTBot and still allow OAI-SearchBot to stay eligible for citations.
No. Google-Extended only controls whether your content is used for Gemini training and grounding. Google Search, including AI Overviews, is crawled by Googlebot, which Google-Extended does not affect. Your rankings, snippets and eligibility for AI Overviews stay the same.
No. Google stopped supporting noindex in robots.txt in September 2019. Use a meta robots noindex tag or an X-Robots-Tag header, and make sure the page is not blocked in robots.txt, or Google will never see the tag.
Yes. The Sitemap line must be a full URL starting with https://. A relative path like /sitemap.xml is ignored. This tool converts relative sitemap paths using your website address, or removes them if no address is set.
No. It stops crawling, not indexing. A blocked URL that other sites link to can still appear in results without a description. To keep a page out of search, allow crawling and add a noindex tag.
Google ignores priority and changefreq. It uses lastmod when it is consistently accurate. Only set lastmod to the date the content really changed, not the date the sitemap was generated.
robots.txt is a voluntary standard. The major AI companies publish their crawler names and say they respect it, though OpenAI notes user initiated ChatGPT-User fetches may not. For a crawler that ignores it, block it at the server or firewall.
More Free Tools
Built from the same toolkits Wise Media runs client work on.
llms.txt Generator
Build a valid llms.txt for AI search in two minutes.
Use The ToolSchema Generator And Validator
Organization, LocalBusiness, Service, FAQPage and BreadcrumbList JSON-LD.
Use The ToolAnswer Block Checker
Does the page answer its question in the first 60 words?
Use The ToolTitle And Meta Description Preview
See your search snippet, its truncation and its click reason.
Use The ToolService And Area Matrix Builder
Every service by area page, with a warning where content is thin.
Use The ToolDNS Record Generator
SPF, DKIM and DMARC records plus the DMARC policy schedule.
Use The ToolRather Have It
Done For You?
Wise Media builds and runs this for clients. Thirty minutes with Cody: the bottleneck named, the fix scoped, the price on the table.