freecourses.digital
Lesson progress: 0 of 2 activities done
All lessons in AI Search and Answer Engine Optimization Lesson 2.2 of 22

Module 1: Understanding AI Search

  1. 1.1What is AI search?
  2. 1.2Traditional search vs. AI search
  3. 1.3Google AI Overviews, ChatGPT Search, Perplexity, and Microsoft Copilot
  4. 1.4Brand mentions vs. citations in AI answers

Module 2: Technical Accessibility for AI Systems

  1. 2.1AI crawlers and user agents
  2. 2.2Robots.txt and AI crawler access
  3. 2.3Crawling, indexing, and retrieval in AI search
  4. 2.4JavaScript rendering and server-side accessibility
  5. 2.5CDN rules, firewalls, and bot management

Module 3: Creating Content That Can Be Cited

  1. 3.1Writing direct answers to specific questions
  2. 3.2Evidence, original data, and verifiable claims
  3. 3.3Entity clarity and brand recognition
  4. 3.4Content interpretation, context, and ambiguity
  5. 3.5Structured data and its role in AI search

Module 4: Measuring AI Search Visibility

  1. 4.1Measuring brand mentions and source citations
  2. 4.2Identifying AI search referral traffic
  3. 4.3Testing visibility across AI search platforms
  4. 4.4Measurement limitations and reliable reporting

Module 5: Improving AI Search Readiness

  1. 5.1Conducting an AI search readiness audit
  2. 5.2Finding brand, product, and content gaps
  3. 5.3Prioritizing pages for AI search improvement
  4. 5.4Building an AI search monitoring plan

Robots.txt and AI crawler access

Your roadmap for this lesson

  • How robots.txt rules work for AI bots
  • How to allow search bots while blocking training bots
  • How to test your robots.txt safely

Short answer

Robots.txt is a text file at the root of your site that tells crawlers which pages they may visit. You add a User-agent line for each AI bot and an Allow or Disallow rule. To appear in AI search, allow search bots like OAI-SearchBot, PerplexityBot, Googlebot, and Bingbot. You can still block training bots like GPTBot if you prefer. Reputable bots follow these rules; robots.txt is not a security tool.

Where robots.txt lives

Open yoursite.com/robots.txt in a browser. If you see a file, that is your current rule set. On WordPress it is often generated by an SEO plugin like Yoast or Rank Math.

Three common setups

1. Allow all AI bots (best for visibility)

User-agent: *
Allow: /

Sitemap: https://example.ph/sitemap.xml

2. Allow AI search, block AI training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

3. Block private areas for everyone

User-agent: *
Disallow: /my-account/
Disallow: /checkout/

How bots read the file

A bot looks for a group that names it specifically. If it finds one, it follows only that group and ignores the * group. So if you give GPTBot its own group, repeat any private-folder rules inside that group too.

Common mistakes

MistakeResultFix
Disallow: / under User-agent: * from a staging siteEvery bot blocked, including GooglebotRemove it after launch
Blocking all “AI” bots with one plugin toggleSearch bots blocked tooAllow search bots by name
Typo in bot name (OAISearchBot)Rule ignoredCopy names from official docs
Blocking CSS and JS foldersPages render badly for crawlersAllow assets

Example: A Davao resort blocked GPTBot to protect its photos and copied the same block for OAI-SearchBot by mistake. It disappeared from ChatGPT’s hotel suggestions. Splitting the rules restored it.

Test changes in Google Search Console’s robots.txt report and recheck the live file after every plugin update.

Quick quiz

1. A bot finds a group with its own name. What does it do with the * group?
2. Is robots.txt a security tool?

Your action

Key terms

Robots.txt
A file that gives crawl instructions to bots.
Disallow
A rule telling a bot not to crawl a path.
Allow
A rule permitting a bot to crawl a path.
Wildcard user agent
The * line that applies to all bots without their own group.

Questions beginners ask

Does blocking in robots.txt remove pages already used?

No. It stops future crawling. Content already collected may remain in models or indexes for some time.

What about llms.txt?

It is a proposed file that lists your key pages for AI tools. It is cheap to add, but no major AI search platform has confirmed using it for ranking or citations, so treat it as optional.

Related lessons

Want someone to audit your AI search visibility while you learn? The team at Remar Digital can help.