Robots.txt

Robots.txt is a plain text file at the root of a host, such as https://example.com/robots.txt. It only applies to that exact host and protocol, so a subdomain like shop.example.com needs its own file. Google reads up to 500 KiB of it and ignores anything beyond that.

The file is built from groups. Each group starts with a User-agent line naming a crawler, or an asterisk for all crawlers, followed by Disallow and Allow rules for URL paths. When rules conflict, Google follows the most specific one, meaning the longest matching path. A Sitemap line can sit anywhere in the file and points crawlers to your XML sitemap. Google ignores crawl-delay, and it stopped supporting noindex inside robots.txt in 2019.

The rule most people get wrong: Disallow blocks crawling, not indexing. A blocked URL can still appear in search results, without a description, if other pages link to it. To keep a page out of the index, allow crawling and use a noindex tag.

AI companies now split their bots into three jobs, each with its own user-agent token. Search crawlers index pages so they can be cited in AI answers: OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), and PerplexityBot. Training crawlers collect content for future models: GPTBot (OpenAI), ClaudeBot (Anthropic), Applebot-Extended, Meta-ExternalAgent, and CCBot (Common Crawl). User fetchers load a page live when someone asks the assistant about it: ChatGPT-User, Claude-User, and Perplexity-User. Some vendors state that user-initiated fetches may not follow robots.txt.

This split lets a site stay citable while opting out of training: allow the search crawlers and disallow the training ones. Google-Extended works differently. It is a control token, not a separate crawler, and it decides whether Google can use your content for Gemini models. Blocking it does not affect Google Search or AI Overviews.