Myths & debunks
Are You Blocking ChatGPT by Accident?
Sometime in 2024 or 2025, advice spread across the web: block AI scrapers. Protect your content. Don’t let OpenAI train on your work for free.
If you followed that advice, go check your robots.txt before you read any further.
You may have blocked the wrong crawler.
GPTBot and OAI-SearchBot are two different things
OpenAI runs two distinct web crawlers, and they do completely different jobs.
GPTBot reads your content to train OpenAI’s language models — the underlying AI itself. When GPTBot visits your site, it’s collecting text for future model training. Blocking GPTBot prevents OpenAI from using your content that way. That’s a legitimate thing to care about, and it has zero effect on whether your business appears in ChatGPT’s search results today.
OAI-SearchBot does something different. It’s OpenAI’s live search crawler — the bot that powers ChatGPT’s real-time web search. When ChatGPT searches the web to answer a question, OAI-SearchBot is what crawls pages in real time. Block OAI-SearchBot, and your site disappears from ChatGPT’s live citations.
Both are documented in OpenAI’s bot documentation. They have different User-agent strings, different functions, and completely different implications for your business. Blocking one does nothing to the other.
How the mistake happens
The “block AI scrapers” advice that spread in 2024–25 usually targeted GPTBot specifically — because GPTBot was the crawler that made the news. It was named in legal filings, discussed in developer forums, written about by major publishers. OAI-SearchBot came later, once ChatGPT’s live search feature launched.
The problem is how crawler blocks get written. A Disallow: / rule under User-agent: * — the wildcard that means “all crawlers” — blocks everything, including OAI-SearchBot. Security plugins, hosting platforms, and WordPress tools that added broad AI-crawler blocks often applied them before OAI-SearchBot existed as a named user-agent.
Cloudflare Radar tracks AI bot activity across the web, and GPTBot consistently shows up as one of the most widely blocked AI crawlers. Many of those blocks were written broadly. If your site’s crawler rules haven’t been reviewed since early 2024, there’s a real chance the update that was supposed to protect you also cut you off from ChatGPT’s live search.
Check your robots.txt right now (90 seconds)
Go to yourdomain.com/robots.txt in your browser.
Look for one of these three situations:
1. An explicit OAI-SearchBot block:
User-agent: OAI-SearchBot
Disallow: /If you see this, you’ve explicitly blocked ChatGPT’s live search crawler.
2. A catch-all block with no allowlist:
User-agent: *
Disallow: /If this is the only rule in your file with no specific Allow: exception for OAI-SearchBot, you’ve blocked it by default.
3. No mention of OAI-SearchBot at all:
If OAI-SearchBot doesn’t appear in your robots.txt, ChatGPT’s crawler can access your site. This is the most common situation. You’re not blocking yourself.
If you need to fix a block: remove the OAI-SearchBot disallow rule, or add an explicit Allow: / exception for it. You can block GPTBot and allow OAI-SearchBot in the same file — they’re separate rules. OpenAI’s documentation lists both user-agent strings.
Robots.txt is a floor condition — not a ceiling
Robots.txt is the first thing we check in an AI visibility audit, and it’s one of the fastest fixes when something’s wrong.
But fixing it doesn’t guarantee that AI engines will cite your business. In the six audits we’ve run, robots.txt blocks were rarely the core problem. The more common finding: a site that’s technically crawlable but barely mentioned across the sources that AI engines actually read — directories, review platforms, local news, industry sites. The rest of the AI Visibility Myths cluster covers the other common misconceptions: whether llms.txt changes anything, and whether GEO is just SEO under a new name.
When we ran an AI visibility audit on our own business, zero of 100 engine responses linked to us — even with a clean robots.txt, even for searches where someone was already looking for a company like ours. Here’s exactly what we found and what was missing.
→ We scored ourselves 0/100 on citations. Read the full audit.
→ Get a free AI Visibility Snapshot — we’ll check your site’s crawler access as part of it.
Data from OpenAI’s bot documentation, current as of August 2026. AI crawler policies change frequently — if you’re reading this more than six months after the publication date above, check OpenAI’s bot documentation for current user-agent strings and policies.