The robots.txt mistake that hides you from AI.

Your website can be public, indexed by Google and still tell important AI-related crawlers not to enter.

The instruction usually sits in a small text file at the root of the site:

https://example.com/robots.txt

iSeer's crawler diagnostic can run this check for a tracked brand and map a detected block to the exact fix template.

The fix can take ten minutes. Deciding whether you want the fix is a policy choice.

What robots.txt does.

robots.txt gives instructions to automated crawlers. A rule can apply to every crawler or to a named user agent.

This rule blocks all compliant crawlers from the whole site:

User-agent: *
Disallow: /

A more targeted rule blocks one named crawler:

User-agent: CCBot
Disallow: /

Common Crawl confirms that CCBot identifies itself as CCBot and obeys robots.txt. OpenAI and Anthropic publish their own crawler controls and user-agent documentation.

A block is not inherently a mistake. Publishers may have legal, licensing, privacy, cost or commercial reasons to restrict automated access.

The mistake is blocking a crawler unintentionally while expecting the content to be broadly available to AI systems.

Three names you may find.

GPTBot

GPTBot is OpenAI's documented web crawler, associated with content that may be used to improve OpenAI's models. OpenAI also documents separate agents — such as OAI-SearchBot and ChatGPT-User — for search and user-triggered fetching, so one token does not control every OpenAI surface. Verify the current tokens in OpenAI's bot documentation.

ClaudeBot

ClaudeBot is Anthropic's documented web crawler. Anthropic states that it respects robots.txt. Verify the current token in Anthropic's help documentation.

CCBot

CCBot is Common Crawl's crawler. Common Crawl publishes a large open web corpus used by many researchers and downstream systems. Common Crawl states that CCBot obeys robots.txt.

Blocking one of these agents does not prove that your company will disappear from ChatGPT or Claude. Model answers can draw on multiple sources, retrieval systems and previously acquired information.

The defensible claim is narrower: a disallow rule can prevent that crawler from accessing the blocked paths, reducing one route by which your public information may be collected or refreshed.

The accidental block.

The common failure is a broad rule added for another reason:

User-agent: *
Disallow: /

This sometimes survives from a staging environment after launch.

Another version blocks a collection of AI-related agents copied from a template without anyone deciding whether that policy matches the company's goals:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

The syntax is simple. The consequence is easy to miss because the site still works for human visitors.

A permissive starting point.

To permit compliant crawlers across the public site, a minimal file can be:

User-agent: *
Allow: /

Many sites do not need an explicit Allow: /; an empty or absent disallow rule can also permit crawling. Existing search, security and platform requirements should be preserved.

To block private areas while permitting public pages:

User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /checkout/
Allow: /

Do not paste this blindly. Paths, authentication and crawler behaviour differ by site. robots.txt is not an access-control system. Sensitive pages must be protected by authentication, not by a request to crawlers.

Allowing selected crawlers.

A site may choose a general policy with explicit exceptions. The exact behaviour of conflicting rules should be tested against the crawlers concerned.

A straightforward explicit allowance looks like this:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: CCBot
Allow: /

Check this snippet against the current official documentation for all three agents before deploying it. OpenAI's product-specific crawlers distinguish training, search and user-triggered access; one token does not control every OpenAI surface.

Check more than the homepage.

A crawler may be allowed to fetch / but blocked from the pages that explain the product.

Inspect rules affecting:

• product pages;
• pricing;
• documentation;
• comparison pages;
• customer stories with recorded consent;
• category pages;
• the sitemap.

Also check whether a content-delivery network, firewall or bot-management service blocks the crawler even when robots.txt allows it.

A clean file does not guarantee successful access. It only removes one explicit instruction not to crawl.

What iSeer checks.

iSeer's crawler diagnostic inspects the site and maps a detected problem to a specific remediation template. The findings it reports include:

{N} AI crawler(s) are disallowed in robots.txt
No robots.txt was found at your domain
{UA} is being blocked by your CDN

The {N} and {UA} placeholders are filled in per site — the count of blocked crawlers and the specific user agent concerned.

The check is diagnostic, not magical. It can identify that a crawler is disallowed and provide the corresponding fix template. It cannot guarantee that allowing the crawler will cause a model to recommend the brand.

Recommendation depends on category recognition, positioning, evidence, platform behaviour and repeated measurement.

The ten-minute procedure.

Open /robots.txt. Search for GPTBot, ClaudeBot, CCBot and User-agent: *.

For every Disallow, decide whether it is intentional.

Then:

1. remove accidental broad blocks;
2. preserve restrictions for private and operational paths;
3. deploy the file;
4. fetch it from the public domain;
5. verify that the relevant public pages are allowed;
6. re-run the crawler diagnostic.

Keep the change in version control. A robots policy is part of the site's publishing policy and should not live as an unexplained server edit.

Fix access, then measure the result.

Changing robots.txt is an infrastructure correction. It is not a visibility result.

After the fix, allow time for crawling and downstream systems to change. Then measure the same buyer-intent questions again across ChatGPT and Claude.

If the brand remains absent, inspect the other failure modes described in Why ChatGPT doesn't recommend you. The methodology explains how iSeer separates repeated evidence from one-run noise. a category leaderboard can show whether the problem affects your brand alone or the way the category is represented.

A crawler block is useful to find because it is concrete. It is rarely the whole story.