freecourses.digital
Lesson progress: 0 of 3 activities done
All lessons in Technical SEO Auditing Lesson 2.3 of 84

Module 1: Introduction to Technical SEO Auditing

  1. 1.1What Is a Technical SEO Audit?
  2. 1.2Technical SEO Audit vs. General SEO Audit
  3. 1.3When Should You Audit a Website?
  4. 1.4Types of Technical SEO Issues
  5. 1.5Defining Audit Scope and Objectives

Module 2: Website Crawling and URL Discovery

  1. 2.1How SEO Crawlers Discover URLs
  2. 2.2Configuring a Website Crawl
  3. 2.3Crawl Scope, Limits, and Exclusions
  4. 2.4Internal Links and Crawl Depth
  5. 2.5Orphan Pages and URL Discovery
  6. 2.6Crawl Data vs. Search Engine Data
  7. 2.7Identifying Crawl Coverage Gaps

Module 3: HTTP Status Codes and Redirect Auditing

  1. 3.1Understanding HTTP Status Codes
  2. 3.2Finding 4xx Errors and Broken Links
  3. 3.3Identifying 5xx Server Errors
  4. 3.4Redirect Chains and Redirect Loops
  5. 3.5Internal Links to Redirected URLs
  6. 3.6301 vs. 302 Redirects
  7. 3.7Auditing 404 Pages and Soft 404s

Module 4: Crawlability and Indexability Auditing

  1. 4.1Crawling vs. Indexing
  2. 4.2Robots.txt Audit
  3. 4.3Meta Robots and X-Robots-Tag
  4. 4.4Noindex and Canonical Conflicts
  5. 4.5XML Sitemap Validation
  6. 4.6Canonical URL Auditing
  7. 4.7Identifying Indexability Conflicts
  8. 4.8Comparing Crawl Data with Google Search Console

Module 5: URL Structure and Duplicate Content Issues

  1. 5.1URL Structure and Normalization
  2. 5.2Duplicate URLs and Duplicate Pages
  3. 5.3HTTP vs. HTTPS and Hostname Variations
  4. 5.4Trailing Slashes and URL Parameters
  5. 5.5Pagination and Faceted Navigation
  6. 5.6Canonicalization Strategies
  7. 5.7Internal Linking to Preferred URLs

Module 6: Technical On-Page Auditing

  1. 6.1Missing and Duplicate Title Tags
  2. 6.2Missing and Duplicate Meta Descriptions
  3. 6.3Missing, Duplicate, and Multiple H1 Headings
  4. 6.4Heading Hierarchy and HTML Structure
  5. 6.5Missing Image Alt Attributes
  6. 6.6Internal Link and Anchor Text Issues
  7. 6.7Empty, Thin, and Near-Duplicate Pages
  8. 6.8Auditing HTML Metadata at Scale

Module 7: Website Performance and Core Web Vitals

  1. 7.1Core Web Vitals: LCP, INP, and CLS
  2. 7.2Lab Data vs. Field Data
  3. 7.3Identifying Slow Pages
  4. 7.4Image and Resource Optimization Opportunities
  5. 7.5JavaScript and CSS Performance
  6. 7.6Mobile Usability and Responsive Rendering
  7. 7.7Prioritizing Performance Fixes

Module 8: JavaScript SEO and Rendering

  1. 8.1How Search Engines Process JavaScript
  2. 8.2HTML Source vs. Rendered HTML
  3. 8.3Client-Side vs. Server-Side Rendering
  4. 8.4JavaScript-Generated Content and Links
  5. 8.5Rendering Failures and Missing Content
  6. 8.6Lazy Loading and Crawl Accessibility
  7. 8.7Diagnosing JavaScript SEO Problems

Module 9: Structured Data and Schema Auditing

  1. 9.1Structured Data and Schema.org Basics
  2. 9.2JSON-LD and Structured Data Formats
  3. 9.3Finding Structured Data Across a Website
  4. 9.4Detecting Invalid or Incomplete Markup
  5. 9.5Schema Validation and Rich Result Eligibility
  6. 9.6Conflicting or Duplicate Structured Data
  7. 9.7Prioritizing Schema Fixes

Module 10: Analyzing Audit Data and Prioritizing Issues

  1. 10.1Reading Crawl Reports and Exports
  2. 10.2Grouping URLs by Issue Type
  3. 10.3Identifying Root Causes vs. Symptoms
  4. 10.4Evaluating Severity, Impact, and Effort
  5. 10.5Separating Sitewide and Isolated Issues
  6. 10.6Identifying False Positives
  7. 10.7Building an Actionable Issue Register

Module 11: Technical SEO Audit Reporting

  1. 11.1Structuring a Technical SEO Audit Report
  2. 11.2Documenting Evidence and Affected URLs
  3. 11.3Writing Developer-Ready Recommendations
  4. 11.4Mapping Issues to Business and Search Impact
  5. 11.5Creating a Prioritized Remediation Roadmap
  6. 11.6Presenting Findings to Stakeholders
  7. 11.7Tracking Fixes and Ownership

Module 12: Fix Verification and Technical QA

  1. 12.1Re-Crawling After Fixes
  2. 12.2Comparing Before-and-After Crawl Results
  3. 12.3Verifying Redirects, Canonicals, and Indexability
  4. 12.4Detecting Regressions and Newly Introduced Issues
  5. 12.5Testing Website Changes Before Deployment
  6. 12.6Confirming Fixes with Search Console and Other Evidence
  7. 12.7Closing Audit Findings and Recording Results

Crawl Scope, Limits, and Exclusions

What we will go through

  • How to include or exclude subdomains and folders
  • When to use URL, depth, and parameter limits
  • How to write exclusion rules with regex patterns
  • How to avoid excluding the problems you need to find

Short answer

Crawl scope controls which URLs your crawler visits. Limits cap total URLs, crawl depth, and query parameters, while exclusions skip patterns like admin pages, cart URLs, and endless filter combinations. Good scope settings keep crawls fast and focused on pages that matter for search, without hiding the problems you need to find.

A Manila fashion shop has 3,000 products. Its filters (size, color, price, sort) can combine into more than 4 million URLs. Crawl it with no limits and your laptop runs out of memory before you reach the blog. Scope settings fix this, but used carelessly they also hide real problems.

The three controls

1. Scope (where to crawl). Choose subdomains and folders. Crawl only www., or add shop. and blog. if your audit brief includes them. Use include rules to stay inside a folder.

2. Limits (how much to crawl). Common limits:

  • Total URLs, such as 100,000
  • Crawl depth, such as 10 clicks
  • Query parameters per URL, such as 2
  • Folder depth or URL length

3. Exclusions (what to skip). Use regex patterns for URLs that add no audit value:

  • .*/wp-admin/.* (admin area)
  • .*\?add-to-cart=.* (cart actions)
  • .*/calendar/\d{4}/.* (endless calendar pages)

Sample first, then exclude

Do not exclude parameter or filter URLs blindly. Let the first crawl run with a limit, say 20,000 URLs, and look at what it finds. If 15,000 URLs are filter combinations, that is a finding: Google may waste crawl effort the same way. Document it, then exclude those patterns in the next crawl so you can audit the real pages.

Spotting crawl traps

Signs your crawl is stuck in a trap:

  • URL count rises fast while unique titles stay flat
  • URLs get longer and longer, like /a/b/a/b/a/b/
  • One folder holds most of the crawl

Pause, find the pattern, and add an exclusion.

Practice: exclude or keep?

Sort each item into the right group.

  • /wp-admin/ pages

  • Product pages under /shop/

  • ?add-to-cart= action URLs

  • A sample of ?color= filter URLs on the first crawl

  • Calendar pages for the year 2087

  • Blog category pages

Common mistakes

  • Excluding problems before documenting them. Parameter bloat is often the biggest finding.
  • Setting URL limits too low. The crawl stops halfway and you report a partial site.
  • Forgetting HTTP or non-www variants. You need them to check redirects in Module 3 and 5.
  • Writing regex that matches too much. Test patterns on a few URLs first.
1. A 2,000-page site produces 80,000 crawled URLs. What does that suggest?
2. What should you do with filter URLs on the first crawl?
3. Do crawler exclusions stop Google from crawling those URLs?

Your action

Next, you will learn how internal links and crawl depth shape what search engines find first.

Key terms

Include rule
A pattern that restricts the crawl to matching URLs only, such as /blog/.
Exclude rule
A pattern that tells the crawler to skip matching URLs, such as anything containing ?add-to-cart=.
Regex
A regular expression, a short text pattern used to match many URLs with one rule, like .*\?sort=.*
Crawl trap
A URL pattern that creates endless new URLs, such as calendars or stacked filters, which can stall a crawl.

Questions beginners ask

Should I exclude URL parameters from every crawl?

No. The first crawl should include a sample so you can see whether parameters create duplicate or crawlable URL problems. Exclude or limit them after you have documented the issue.

What is a sensible URL limit?

Use your best estimate of real pages plus about 50%. If a 2,000-page site produces 80,000 URLs, that gap itself is a finding.

How do I crawl only one subfolder?

Start the crawl from that folder's URL and use an include rule like https://www.example.ph/shop/.*

Do exclusions affect Google?

No. They only change your crawler's behavior. To control Google, you use robots.txt, noindex, or canonicals.

Related lessons

Want someone to handle SEO for your business while you learn? The team at Remar Digital can help.