freecourses.digital
Lesson progress: 0 of 3 activities done
All lessons in Technical SEO Auditing Lesson 4.2 of 84

Module 1: Introduction to Technical SEO Auditing

  1. 1.1What Is a Technical SEO Audit?
  2. 1.2Technical SEO Audit vs. General SEO Audit
  3. 1.3When Should You Audit a Website?
  4. 1.4Types of Technical SEO Issues
  5. 1.5Defining Audit Scope and Objectives

Module 2: Website Crawling and URL Discovery

  1. 2.1How SEO Crawlers Discover URLs
  2. 2.2Configuring a Website Crawl
  3. 2.3Crawl Scope, Limits, and Exclusions
  4. 2.4Internal Links and Crawl Depth
  5. 2.5Orphan Pages and URL Discovery
  6. 2.6Crawl Data vs. Search Engine Data
  7. 2.7Identifying Crawl Coverage Gaps

Module 3: HTTP Status Codes and Redirect Auditing

  1. 3.1Understanding HTTP Status Codes
  2. 3.2Finding 4xx Errors and Broken Links
  3. 3.3Identifying 5xx Server Errors
  4. 3.4Redirect Chains and Redirect Loops
  5. 3.5Internal Links to Redirected URLs
  6. 3.6301 vs. 302 Redirects
  7. 3.7Auditing 404 Pages and Soft 404s

Module 4: Crawlability and Indexability Auditing

  1. 4.1Crawling vs. Indexing
  2. 4.2Robots.txt Audit
  3. 4.3Meta Robots and X-Robots-Tag
  4. 4.4Noindex and Canonical Conflicts
  5. 4.5XML Sitemap Validation
  6. 4.6Canonical URL Auditing
  7. 4.7Identifying Indexability Conflicts
  8. 4.8Comparing Crawl Data with Google Search Console

Module 5: URL Structure and Duplicate Content Issues

  1. 5.1URL Structure and Normalization
  2. 5.2Duplicate URLs and Duplicate Pages
  3. 5.3HTTP vs. HTTPS and Hostname Variations
  4. 5.4Trailing Slashes and URL Parameters
  5. 5.5Pagination and Faceted Navigation
  6. 5.6Canonicalization Strategies
  7. 5.7Internal Linking to Preferred URLs

Module 6: Technical On-Page Auditing

  1. 6.1Missing and Duplicate Title Tags
  2. 6.2Missing and Duplicate Meta Descriptions
  3. 6.3Missing, Duplicate, and Multiple H1 Headings
  4. 6.4Heading Hierarchy and HTML Structure
  5. 6.5Missing Image Alt Attributes
  6. 6.6Internal Link and Anchor Text Issues
  7. 6.7Empty, Thin, and Near-Duplicate Pages
  8. 6.8Auditing HTML Metadata at Scale

Module 7: Website Performance and Core Web Vitals

  1. 7.1Core Web Vitals: LCP, INP, and CLS
  2. 7.2Lab Data vs. Field Data
  3. 7.3Identifying Slow Pages
  4. 7.4Image and Resource Optimization Opportunities
  5. 7.5JavaScript and CSS Performance
  6. 7.6Mobile Usability and Responsive Rendering
  7. 7.7Prioritizing Performance Fixes

Module 8: JavaScript SEO and Rendering

  1. 8.1How Search Engines Process JavaScript
  2. 8.2HTML Source vs. Rendered HTML
  3. 8.3Client-Side vs. Server-Side Rendering
  4. 8.4JavaScript-Generated Content and Links
  5. 8.5Rendering Failures and Missing Content
  6. 8.6Lazy Loading and Crawl Accessibility
  7. 8.7Diagnosing JavaScript SEO Problems

Module 9: Structured Data and Schema Auditing

  1. 9.1Structured Data and Schema.org Basics
  2. 9.2JSON-LD and Structured Data Formats
  3. 9.3Finding Structured Data Across a Website
  4. 9.4Detecting Invalid or Incomplete Markup
  5. 9.5Schema Validation and Rich Result Eligibility
  6. 9.6Conflicting or Duplicate Structured Data
  7. 9.7Prioritizing Schema Fixes

Module 10: Analyzing Audit Data and Prioritizing Issues

  1. 10.1Reading Crawl Reports and Exports
  2. 10.2Grouping URLs by Issue Type
  3. 10.3Identifying Root Causes vs. Symptoms
  4. 10.4Evaluating Severity, Impact, and Effort
  5. 10.5Separating Sitewide and Isolated Issues
  6. 10.6Identifying False Positives
  7. 10.7Building an Actionable Issue Register

Module 11: Technical SEO Audit Reporting

  1. 11.1Structuring a Technical SEO Audit Report
  2. 11.2Documenting Evidence and Affected URLs
  3. 11.3Writing Developer-Ready Recommendations
  4. 11.4Mapping Issues to Business and Search Impact
  5. 11.5Creating a Prioritized Remediation Roadmap
  6. 11.6Presenting Findings to Stakeholders
  7. 11.7Tracking Fixes and Ownership

Module 12: Fix Verification and Technical QA

  1. 12.1Re-Crawling After Fixes
  2. 12.2Comparing Before-and-After Crawl Results
  3. 12.3Verifying Redirects, Canonicals, and Indexability
  4. 12.4Detecting Regressions and Newly Introduced Issues
  5. 12.5Testing Website Changes Before Deployment
  6. 12.6Confirming Fixes with Search Console and Other Evidence
  7. 12.7Closing Audit Findings and Recording Results

Robots.txt Audit

In this lesson

  • How robots.txt rules are written and matched
  • What to check during a robots.txt audit
  • How to find important pages and resources that are blocked
  • How to test changes before publishing

Short answer

A robots.txt audit checks that the file at your domain root allows crawling of important pages, CSS, and JavaScript, while blocking only low-value areas like carts and internal search. Read each rule, list blocked URLs from your crawl, test them in Search Console's robots.txt report, and confirm sitemap lines point to live sitemaps.

One line can hide a whole website: Disallow: /. It is the default on many staging sites, and it sometimes survives launch day. That is why robots.txt is one of the first files every auditor opens.

Reading the file

User-agent: *
Disallow: /cart/
Disallow: /search
Disallow: /*?sort=
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/

Sitemap: https://www.example.ph/sitemap_index.xml
  • Rules are grouped by User-agent.
  • Paths match from the start of the URL. /search blocks /search, /search/, and /searchresults.
  • * matches anything, $ marks the end.
  • Google follows the most specific matching rule, so Allow can open a path inside a Disallow.

The audit checklist

  1. File status: robots.txt returns 200 (or 404). A 5xx is urgent.
  2. No sitewide block: no Disallow: / for Googlebot or *.
  3. Important sections allowed: products, services, blog, locations.
  4. Resources allowed: CSS, JavaScript, images, fonts.
  5. Low-value areas blocked on purpose: carts, checkout, internal search, endless filters.
  6. Sitemap lines point to live sitemaps on the right host.
  7. Every host checked: www, non-www, subdomains, staging.

Finding blocked URLs

Run your crawl with “respect robots.txt” on, then open the Blocked by Robots.txt filter. For each, the crawler shows the matching rule. Compare with Search Console’s robots.txt report and the “Blocked by robots.txt” reason in the Pages report.

Practice: should this be blocked?

Sort each item into the right group.

  • /checkout/

  • /wp-content/themes/ (CSS and JS)

  • Internal search results /search?q=

  • /products/

  • Endless sort parameters /*?sort=

  • /blog/

Common mistakes

  • Staging Disallow: / pushed live.
  • Blocking CSS or JS folders. Pages render broken for Google.
  • Using robots.txt to hide pages from results. Use noindex instead (Lesson 4.3).
  • Overly broad patterns. Disallow: /p blocks /products/ too.
1. What does Disallow: /p block?
2. Why should CSS and JS stay crawlable?
3. Robots.txt returns 503 for days. What may happen?

Your action

Next, you will audit meta robots tags and X-Robots-Tag headers.

Key terms

User-agent
The line naming which crawler a group of rules applies to, such as * for all or Googlebot.
Disallow
A rule telling crawlers not to fetch URLs that start with or match a path.
Allow
A rule that permits a path inside a broader Disallow. Google uses the most specific matching rule.
Wildcard
The * symbol matching any characters, and $ marking the end of a URL, used in patterns like /*?sort=

Questions beginners ask

Where is robots.txt located?

Always at the root of each host, like https://www.example.ph/robots.txt. Each subdomain and protocol needs its own file.

Should I block CSS and JavaScript?

No. Google needs them to render pages. Blocking them can make pages look broken or empty to Googlebot.

What happens if robots.txt returns 5xx?

Google may pause crawling the whole site until it can read the file. A 404 robots.txt is treated as no rules, meaning everything is allowed.

Should I block AI crawlers in robots.txt?

That is a business choice. Check which user agents are listed and confirm the decision is intentional, not copied from a template.

Related lessons

Want someone to handle SEO for your business while you learn? The team at Remar Digital can help.