FreeYour first AI visibility check in 5 minutes, refreshed every 7 daysTry it free
GEO Scout
Get started
Free tool

Check robots.txt

Enter a website: we check robots.txt for errors and show who can read the site — search engines, ChatGPT, Claude and Perplexity AI search, AI agents and model training bots. For blocked AI bots we prepare ready rules.

Free, no sign-up. We read /robots.txt, the homepage and sitemaps, which takes up to 15 seconds.

Which bots we check

Four groups of bots, each with its own impact

You cannot block “AI” with one line: each company runs several bots with different jobs. Some fetch pages for answers, some collect texts for training, some carry out user tasks.

18 bots

AI search

Read pages at the moment an AI engine answers. Block them and ChatGPT, Claude and Perplexity stop citing the site.

OAI-SearchBot · ChatGPT-User · Claude-SearchBot · PerplexityBot

9 bots

Search engines

Classic search. Google AI Overviews and Yandex AI answers rely on these indexes, so these bots matter for AI answers too.

YandexBot · Googlebot · Bingbot

28 bots

AI training

Collect texts to train models. Blocking them does not remove the site from search-based answers, so decide by your content policy.

GPTBot · ClaudeBot · Google-Extended · CCBot

7 bots

AI agents

Act on behalf of a user: compare offers, fill in forms, place orders.

ChatGPT Agent · Operator · Gemini-Deep-Research

How to set it up

How to configure robots.txt for search and AI bots

  1. 01

    Keep search open

    No Disallow: / for Googlebot and YandexBot, otherwise the site drops out of both search and AI Overviews.

  2. 02

    Open AI search

    OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot need access to the pages you want to see in answers.

  3. 03

    Decide on training

    GPTBot, ClaudeBot and Google-Extended can be blocked if you do not want your texts used for training. It does not affect citations in search-based answers.

  4. 04

    List your sitemap

    A Sitemap: line with the full URL helps bots find new pages faster.

  5. 05

    Check again

    Run the check again after editing. robots.txt is a recommendation: hard blocking needs server or CDN rules.

Example: AI search open, training blocked

User-agent: *
Allow: /

# Model training, optional
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# AI search, keep open
User-agent: OAI-SearchBot
Allow: /

Sitemap: https://example.com/sitemap.xml

Rules for a specific bot replace the general User-agent: * group, so service sections closed for everyone must be repeated in that bot group.

Learn more

FAQ

robots.txt questions

How do I check a site robots.txt?
Enter the website in the form above: the tool reads /robots.txt and shows line errors, sitemap availability and the status of every search and AI bot. For search engines, also check the file in Google Search Console.
How do I block AI from my site?
Add Disallow: / for the bots you want to stop, such as GPTBot, ClaudeBot and Google-Extended, which collect training data. Block AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) only if you do not want the site cited in answers.
If I block GPTBot, will my site disappear from ChatGPT?
No. GPTBot collects data to train OpenAI models. ChatGPT search answers rely on OAI-SearchBot and ChatGPT-User, and while they are allowed the site can appear in answers. More in OAI-SearchBot, GPTBot and robots.txt.
Does robots.txt affect Google AI Overviews?
Yes, through the Google index: AI Overviews are built on indexed pages. If Googlebot is blocked, the site appears neither in search nor in AI Overviews. Google-Extended does not control AI Overviews.
Where should robots.txt be?
Only at the domain root: your-site.com/robots.txt. Each subdomain needs its own file. The server should return it with status 200 as plain text.
Why does the tool say scripts and styles are blocked?
robots.txt has Disallow rules for build folders such as /_next/, /assets/ or /wp-includes/. Without JS and CSS a bot sees an empty page, and it gradually drops out of the index. The tool suggests Allow rules to fix it.
Do all bots respect robots.txt?
No, robots.txt is a recommendation. Major companies follow it, but some bots ignore it. For hard blocking use server or CDN rules, for example Cloudflare bot management.
Is it free?
Yes, without sign-up. Up to 10 checks per 10 minutes from one address. We check up to 5 sitemaps listed in robots.txt.