freecourses.digital
Lesson progress: 0 of 3 activities done
All lessons in Technical SEO Auditing Lesson 4.3 of 84

Module 1: Introduction to Technical SEO Auditing

  1. 1.1What Is a Technical SEO Audit?
  2. 1.2Technical SEO Audit vs. General SEO Audit
  3. 1.3When Should You Audit a Website?
  4. 1.4Types of Technical SEO Issues
  5. 1.5Defining Audit Scope and Objectives

Module 2: Website Crawling and URL Discovery

  1. 2.1How SEO Crawlers Discover URLs
  2. 2.2Configuring a Website Crawl
  3. 2.3Crawl Scope, Limits, and Exclusions
  4. 2.4Internal Links and Crawl Depth
  5. 2.5Orphan Pages and URL Discovery
  6. 2.6Crawl Data vs. Search Engine Data
  7. 2.7Identifying Crawl Coverage Gaps

Module 3: HTTP Status Codes and Redirect Auditing

  1. 3.1Understanding HTTP Status Codes
  2. 3.2Finding 4xx Errors and Broken Links
  3. 3.3Identifying 5xx Server Errors
  4. 3.4Redirect Chains and Redirect Loops
  5. 3.5Internal Links to Redirected URLs
  6. 3.6301 vs. 302 Redirects
  7. 3.7Auditing 404 Pages and Soft 404s

Module 4: Crawlability and Indexability Auditing

  1. 4.1Crawling vs. Indexing
  2. 4.2Robots.txt Audit
  3. 4.3Meta Robots and X-Robots-Tag
  4. 4.4Noindex and Canonical Conflicts
  5. 4.5XML Sitemap Validation
  6. 4.6Canonical URL Auditing
  7. 4.7Identifying Indexability Conflicts
  8. 4.8Comparing Crawl Data with Google Search Console

Module 5: URL Structure and Duplicate Content Issues

  1. 5.1URL Structure and Normalization
  2. 5.2Duplicate URLs and Duplicate Pages
  3. 5.3HTTP vs. HTTPS and Hostname Variations
  4. 5.4Trailing Slashes and URL Parameters
  5. 5.5Pagination and Faceted Navigation
  6. 5.6Canonicalization Strategies
  7. 5.7Internal Linking to Preferred URLs

Module 6: Technical On-Page Auditing

  1. 6.1Missing and Duplicate Title Tags
  2. 6.2Missing and Duplicate Meta Descriptions
  3. 6.3Missing, Duplicate, and Multiple H1 Headings
  4. 6.4Heading Hierarchy and HTML Structure
  5. 6.5Missing Image Alt Attributes
  6. 6.6Internal Link and Anchor Text Issues
  7. 6.7Empty, Thin, and Near-Duplicate Pages
  8. 6.8Auditing HTML Metadata at Scale

Module 7: Website Performance and Core Web Vitals

  1. 7.1Core Web Vitals: LCP, INP, and CLS
  2. 7.2Lab Data vs. Field Data
  3. 7.3Identifying Slow Pages
  4. 7.4Image and Resource Optimization Opportunities
  5. 7.5JavaScript and CSS Performance
  6. 7.6Mobile Usability and Responsive Rendering
  7. 7.7Prioritizing Performance Fixes

Module 8: JavaScript SEO and Rendering

  1. 8.1How Search Engines Process JavaScript
  2. 8.2HTML Source vs. Rendered HTML
  3. 8.3Client-Side vs. Server-Side Rendering
  4. 8.4JavaScript-Generated Content and Links
  5. 8.5Rendering Failures and Missing Content
  6. 8.6Lazy Loading and Crawl Accessibility
  7. 8.7Diagnosing JavaScript SEO Problems

Module 9: Structured Data and Schema Auditing

  1. 9.1Structured Data and Schema.org Basics
  2. 9.2JSON-LD and Structured Data Formats
  3. 9.3Finding Structured Data Across a Website
  4. 9.4Detecting Invalid or Incomplete Markup
  5. 9.5Schema Validation and Rich Result Eligibility
  6. 9.6Conflicting or Duplicate Structured Data
  7. 9.7Prioritizing Schema Fixes

Module 10: Analyzing Audit Data and Prioritizing Issues

  1. 10.1Reading Crawl Reports and Exports
  2. 10.2Grouping URLs by Issue Type
  3. 10.3Identifying Root Causes vs. Symptoms
  4. 10.4Evaluating Severity, Impact, and Effort
  5. 10.5Separating Sitewide and Isolated Issues
  6. 10.6Identifying False Positives
  7. 10.7Building an Actionable Issue Register

Module 11: Technical SEO Audit Reporting

  1. 11.1Structuring a Technical SEO Audit Report
  2. 11.2Documenting Evidence and Affected URLs
  3. 11.3Writing Developer-Ready Recommendations
  4. 11.4Mapping Issues to Business and Search Impact
  5. 11.5Creating a Prioritized Remediation Roadmap
  6. 11.6Presenting Findings to Stakeholders
  7. 11.7Tracking Fixes and Ownership

Module 12: Fix Verification and Technical QA

  1. 12.1Re-Crawling After Fixes
  2. 12.2Comparing Before-and-After Crawl Results
  3. 12.3Verifying Redirects, Canonicals, and Indexability
  4. 12.4Detecting Regressions and Newly Introduced Issues
  5. 12.5Testing Website Changes Before Deployment
  6. 12.6Confirming Fixes with Search Console and Other Evidence
  7. 12.7Closing Audit Findings and Recording Results

Meta Robots and X-Robots-Tag

This lesson covers

  • The difference between meta robots and X-Robots-Tag
  • The main directives: noindex, nofollow, and others
  • How to find accidental noindex across a site
  • How to audit non-HTML files like PDFs

Short answer

Meta robots tags sit in a page's HTML head, while X-Robots-Tag sends the same directives in the HTTP header, which also works for PDFs and images. Both control indexing with values like noindex and nofollow. Audit them by filtering your crawl for noindex and checking that no important page carries it by mistake.

An Antipolo events venue updated its WordPress theme. The new theme added noindex to all “page” templates, including Weddings, Debuts, and Corporate Events. Within three weeks, those pages vanished from Google. Accidental noindex is one of the most expensive one-line bugs in SEO.

Two places, same rules

In the HTML head:

<meta name="robots" content="noindex, follow">

In the HTTP header:

X-Robots-Tag: noindex

X-Robots-Tag works for any file type, so it is the way to noindex PDFs, images, and feeds. It is also easy to miss because it is invisible in the page source.

Directives you will see

  • noindex: do not show in results
  • nofollow: do not follow links on this page
  • none: same as noindex, nofollow
  • nosnippet, max-snippet, max-image-preview: control how results look
  • unavailable_after: drop the page after a date

How to audit

  1. In your crawl, open Directives and filter Noindex. Your crawler checks both the HTML and the header.
  2. Sort noindexed URLs by folder or template. Ask: should this whole group be hidden?
  3. Cross-check with Search Console’s “Excluded by noindex tag” list.
  4. Look for crawler-specific tags (googlebot) and conflicting tags on the same page.
  5. Check that noindexed URLs are not in your XML sitemap.

Practice: should this page be noindexed?

Sort each item into the right group.

  • Thank-you page after a form

  • Main service page

  • Internal search results

  • Location page for your Naga branch

  • Printable duplicate PDF of a web page

  • Blog post with steady search traffic

Common mistakes

  • Theme or plugin noindex on whole templates.
  • Forgetting the header. Only checking the HTML misses X-Robots-Tag.
  • Noindexed pages still in the sitemap. Mixed signals.
  • Noindex plus robots.txt block. Google never sees the noindex.
1. How do you noindex a PDF?
2. Meta robots says index, X-Robots-Tag says noindex. Result?
3. Where should noindexed URLs not appear?

Your action

Next, you will find conflicts between noindex and canonical tags.

Key terms

Meta robots tag
An HTML tag like <meta name="robots" content="noindex"> giving indexing instructions for that page.
X-Robots-Tag
An HTTP response header carrying robots directives, useful for non-HTML files.
noindex
A directive telling search engines not to show the page in results.
nofollow (page-level)
A directive asking search engines not to follow links on the page. Google treats it as a hint.

Questions beginners ask

What happens if meta robots and X-Robots-Tag disagree?

Google applies the most restrictive directive. If either says noindex, the page is not indexed.

Can I target only Googlebot?

Yes, with <meta name="googlebot" content="noindex">. Check for crawler-specific tags during audits, since they are easy to miss.

Does noindex,follow still pass link value?

For a while. Over time, Google may treat long-term noindexed pages like nofollow and crawl them less.

Can JavaScript add or remove noindex?

It can add it, but if the original HTML has noindex, Google may skip rendering. Never rely on JavaScript to remove a noindex.

Related lessons

Want someone to handle SEO for your business while you learn? The team at Remar Digital can help.