freecourses.digital
Lesson progress: 0 of 2 activities done
All lessons in AI Search and Answer Engine Optimization Lesson 2.5 of 22

Module 1: Understanding AI Search

  1. 1.1What is AI search?
  2. 1.2Traditional search vs. AI search
  3. 1.3Google AI Overviews, ChatGPT Search, Perplexity, and Microsoft Copilot
  4. 1.4Brand mentions vs. citations in AI answers

Module 2: Technical Accessibility for AI Systems

  1. 2.1AI crawlers and user agents
  2. 2.2Robots.txt and AI crawler access
  3. 2.3Crawling, indexing, and retrieval in AI search
  4. 2.4JavaScript rendering and server-side accessibility
  5. 2.5CDN rules, firewalls, and bot management

Module 3: Creating Content That Can Be Cited

  1. 3.1Writing direct answers to specific questions
  2. 3.2Evidence, original data, and verifiable claims
  3. 3.3Entity clarity and brand recognition
  4. 3.4Content interpretation, context, and ambiguity
  5. 3.5Structured data and its role in AI search

Module 4: Measuring AI Search Visibility

  1. 4.1Measuring brand mentions and source citations
  2. 4.2Identifying AI search referral traffic
  3. 4.3Testing visibility across AI search platforms
  4. 4.4Measurement limitations and reliable reporting

Module 5: Improving AI Search Readiness

  1. 5.1Conducting an AI search readiness audit
  2. 5.2Finding brand, product, and content gaps
  3. 5.3Prioritizing pages for AI search improvement
  4. 5.4Building an AI search monitoring plan

CDN rules, firewalls, and bot management

This lesson covers

  • How CDNs and firewalls can block AI bots without warning
  • How to check Cloudflare and plugin settings
  • How to allow verified bots safely

Short answer

A CDN or firewall sits in front of your website and can block bots before robots.txt is even read. Cloudflare, security plugins, and hosting firewalls may block AI crawlers by default or through bot fight, rate limit, or country rules. Check these settings, allow verified AI search bots, and confirm in your logs that they receive normal 200 responses.

Why robots.txt is not the whole story

Your robots.txt can say “Allow” while your CDN returns 403 Forbidden to the same bot. The bot never gets the page, and you get no error message. This is one of the most common hidden causes of zero AI citations.

LayerExampleCan block AI bots?
CDNCloudflare, Fastly, AkamaiYes, through AI crawler and bot settings
Hosting firewallImunify360, ModSecurityYes, through bot or rate rules
Security pluginWordfence, iThemesYes, through “block fake bots” or country blocks
Server config.htaccess, Nginx rulesYes, through user agent rules

Cloudflare checklist

In the Cloudflare dashboard, review the AI crawler controls (shown as AI Crawl Control or Block AI bots, depending on your plan), Bot Fight Mode, custom WAF rules that match user agents, and country blocks. Cloudflare can manage AI crawlers by category, so you can allow search bots while blocking training bots. Wording and menu locations change, so search Cloudflare’s docs for the current name.

Country and rate rules

Many Philippine sites block foreign traffic to stop spam. AI crawlers usually come from US data centers, so a “Philippines only” rule blocks them all. Allow verified bots before country rules run. Similarly, strict rate limits can cut off a crawler halfway through a sitemap.

Confirm with logs

Filter your CDN or server logs by AI user agents and look at status codes. 200 means success. 403 or 429 means blocked or throttled. A challenge page (like a CAPTCHA) also counts as a block, because bots cannot solve it.

Example: A Cavite online pharmacy enabled “Under Attack Mode” during a spam wave and forgot to turn it off. Every bot saw a challenge page for two months. Perplexity citations dropped to zero until the setting was reverted.

Verify bots by IP, not just by name. OpenAI, Anthropic, Perplexity, Google, and Microsoft publish IP ranges so fake bots using their names can still be blocked.

Quick quiz

1. Robots.txt allows OAI-SearchBot, but the CDN returns 403. What happens?
2. Why can a 'Philippines only' firewall rule hurt AI visibility?

Your action

Key terms

CDN
A content delivery network that serves and protects your site.
Web application firewall (WAF)
Rules that filter traffic before it reaches your server.
Verified bot
A bot whose identity is confirmed by its published IP ranges.
Rate limiting
Blocking visitors who request too many pages too fast.
HTTP status code
The server's reply, like 200 OK or 403 Forbidden.

Questions beginners ask

Does Cloudflare block AI bots by default?

Cloudflare offers one-click AI crawler blocking and has made blocking the default for many new domains. Check Security settings in your dashboard to see your current state.

Is it safe to allow AI bots?

Allowing verified bots from known providers is low risk. Keep rate limits for unknown traffic.

Related lessons

Want someone to audit your AI search visibility while you learn? The team at Remar Digital can help.