Free tool · On-page SEO

Robots.txt Generator

Create a robots.txt file that tells crawlers which parts of your site they may visit. Upload it to the root of your domain.

Runs in your browser. Nothing you type is sent to a server.

Default for all crawlers Disallowed paths one per line, applied to all crawlers Allowed paths exceptions inside blocked folders Presets

Block AI training crawlers stops content being collected for model training; does not affect Google or Bing search rankings

Block AI search and assistant bots careful: blocking these can remove your pages from ChatGPT Search, Perplexity, and Claude answers

Sitemap URLs one per line Crawl delay in seconds optional, ignored by Google

robots.txt

robots.txt controls crawling, not indexing. To keep a page out of search results, use a noindex meta tag and let it be crawled.

What the Robots.txt Generator does

A robots.txt file sits at the root of a website, at /robots.txt, and tells crawlers which areas they may visit. It is often used to keep bots out of admin pages, carts, and endless search or filter URLs, so crawlers spend their time on pages that matter.

It controls crawling, not indexing. A blocked page can still appear in Google if other sites link to it. To keep a page out of search results, let it be crawled and add a noindex tag instead. This generator also separates AI crawlers into two groups: training crawlers, which collect content for AI models, and AI search bots, which fetch pages for answers in tools like ChatGPT and Perplexity.

How to use it

Pick a starting pointAllow everything for a live site, or block everything for a staging site. Presets fill in common WordPress and store paths.
List paths to block or allowAdd one path per line. Allowed paths create exceptions inside blocked folders.
Choose AI botsTick training crawlers to opt out of model training. Only tick AI search bots if you accept losing visibility in AI answers.
Download and uploadDownload the file and upload it to your domain root, so it opens at yourdomain.com/robots.txt.

Tips for better results

Never block CSS or JavaScript files. Google needs them to render your pages properly.
Always include your sitemap URL so crawlers can find all your important pages.
After uploading, open the file in a browser and check the robots.txt report in Google Search Console.

Frequently asked questions

Does blocking a page remove it from Google?

No. robots.txt stops crawling, not indexing. Use a noindex meta tag on a crawlable page to remove it from results.

Will blocking AI training bots hurt my Google rankings?

No. Training crawlers like GPTBot and Google-Extended are separate from Googlebot, which handles Google Search.

Does Google follow Crawl-delay?

No. Google ignores Crawl-delay. Some other crawlers, like Bingbot, respect it.