freecourses.digital
Lesson progress: 0 of 3 activities done
All lessons in Technical SEO Auditing Lesson 4.1 of 84

Module 1: Introduction to Technical SEO Auditing

  1. 1.1What Is a Technical SEO Audit?
  2. 1.2Technical SEO Audit vs. General SEO Audit
  3. 1.3When Should You Audit a Website?
  4. 1.4Types of Technical SEO Issues
  5. 1.5Defining Audit Scope and Objectives

Module 2: Website Crawling and URL Discovery

  1. 2.1How SEO Crawlers Discover URLs
  2. 2.2Configuring a Website Crawl
  3. 2.3Crawl Scope, Limits, and Exclusions
  4. 2.4Internal Links and Crawl Depth
  5. 2.5Orphan Pages and URL Discovery
  6. 2.6Crawl Data vs. Search Engine Data
  7. 2.7Identifying Crawl Coverage Gaps

Module 3: HTTP Status Codes and Redirect Auditing

  1. 3.1Understanding HTTP Status Codes
  2. 3.2Finding 4xx Errors and Broken Links
  3. 3.3Identifying 5xx Server Errors
  4. 3.4Redirect Chains and Redirect Loops
  5. 3.5Internal Links to Redirected URLs
  6. 3.6301 vs. 302 Redirects
  7. 3.7Auditing 404 Pages and Soft 404s

Module 4: Crawlability and Indexability Auditing

  1. 4.1Crawling vs. Indexing
  2. 4.2Robots.txt Audit
  3. 4.3Meta Robots and X-Robots-Tag
  4. 4.4Noindex and Canonical Conflicts
  5. 4.5XML Sitemap Validation
  6. 4.6Canonical URL Auditing
  7. 4.7Identifying Indexability Conflicts
  8. 4.8Comparing Crawl Data with Google Search Console

Module 5: URL Structure and Duplicate Content Issues

  1. 5.1URL Structure and Normalization
  2. 5.2Duplicate URLs and Duplicate Pages
  3. 5.3HTTP vs. HTTPS and Hostname Variations
  4. 5.4Trailing Slashes and URL Parameters
  5. 5.5Pagination and Faceted Navigation
  6. 5.6Canonicalization Strategies
  7. 5.7Internal Linking to Preferred URLs

Module 6: Technical On-Page Auditing

  1. 6.1Missing and Duplicate Title Tags
  2. 6.2Missing and Duplicate Meta Descriptions
  3. 6.3Missing, Duplicate, and Multiple H1 Headings
  4. 6.4Heading Hierarchy and HTML Structure
  5. 6.5Missing Image Alt Attributes
  6. 6.6Internal Link and Anchor Text Issues
  7. 6.7Empty, Thin, and Near-Duplicate Pages
  8. 6.8Auditing HTML Metadata at Scale

Module 7: Website Performance and Core Web Vitals

  1. 7.1Core Web Vitals: LCP, INP, and CLS
  2. 7.2Lab Data vs. Field Data
  3. 7.3Identifying Slow Pages
  4. 7.4Image and Resource Optimization Opportunities
  5. 7.5JavaScript and CSS Performance
  6. 7.6Mobile Usability and Responsive Rendering
  7. 7.7Prioritizing Performance Fixes

Module 8: JavaScript SEO and Rendering

  1. 8.1How Search Engines Process JavaScript
  2. 8.2HTML Source vs. Rendered HTML
  3. 8.3Client-Side vs. Server-Side Rendering
  4. 8.4JavaScript-Generated Content and Links
  5. 8.5Rendering Failures and Missing Content
  6. 8.6Lazy Loading and Crawl Accessibility
  7. 8.7Diagnosing JavaScript SEO Problems

Module 9: Structured Data and Schema Auditing

  1. 9.1Structured Data and Schema.org Basics
  2. 9.2JSON-LD and Structured Data Formats
  3. 9.3Finding Structured Data Across a Website
  4. 9.4Detecting Invalid or Incomplete Markup
  5. 9.5Schema Validation and Rich Result Eligibility
  6. 9.6Conflicting or Duplicate Structured Data
  7. 9.7Prioritizing Schema Fixes

Module 10: Analyzing Audit Data and Prioritizing Issues

  1. 10.1Reading Crawl Reports and Exports
  2. 10.2Grouping URLs by Issue Type
  3. 10.3Identifying Root Causes vs. Symptoms
  4. 10.4Evaluating Severity, Impact, and Effort
  5. 10.5Separating Sitewide and Isolated Issues
  6. 10.6Identifying False Positives
  7. 10.7Building an Actionable Issue Register

Module 11: Technical SEO Audit Reporting

  1. 11.1Structuring a Technical SEO Audit Report
  2. 11.2Documenting Evidence and Affected URLs
  3. 11.3Writing Developer-Ready Recommendations
  4. 11.4Mapping Issues to Business and Search Impact
  5. 11.5Creating a Prioritized Remediation Roadmap
  6. 11.6Presenting Findings to Stakeholders
  7. 11.7Tracking Fixes and Ownership

Module 12: Fix Verification and Technical QA

  1. 12.1Re-Crawling After Fixes
  2. 12.2Comparing Before-and-After Crawl Results
  3. 12.3Verifying Redirects, Canonicals, and Indexability
  4. 12.4Detecting Regressions and Newly Introduced Issues
  5. 12.5Testing Website Changes Before Deployment
  6. 12.6Confirming Fixes with Search Console and Other Evidence
  7. 12.7Closing Audit Findings and Recording Results

Crawling vs. Indexing

What we will go through

  • What crawling and indexing each mean
  • Which controls affect crawling and which affect indexing
  • Why a page can be crawled but not indexed
  • How this split shapes the rest of Module 4

Short answer

Crawling is when a search engine fetches a URL to read it. Indexing is when it stores that page so it can appear in results. Robots.txt controls crawling, while noindex and canonical tags guide indexing. A page can be crawled but not indexed, and a blocked page can sometimes be indexed without its content.

A library first has to receive a book before it can put it on a shelf. Receiving is crawling. Shelving it in the catalog is indexing. A book can be received and still not shelved, and a book can appear in the catalog from a review without anyone reading it.

The pipeline

  1. Discover: Google learns the URL exists (links, sitemaps).
  2. Crawl: Googlebot fetches the URL.
  3. Render: Google runs JavaScript to see the final page.
  4. Index: Google decides whether to store it, and which version.
  5. Rank: the indexed page competes in results.

Different controls for different steps

ControlAffectsEffect
robots.txt DisallowCrawlingGooglebot does not fetch the URL
meta robots noindexIndexingPage is dropped from the index (must be crawlable to be seen)
X-Robots-Tag noindexIndexingSame, sent in the HTTP header
rel=canonicalIndexingSuggests which duplicate to index
XML sitemapDiscoverySuggests URLs to crawl

The classic conflict

A Tagaytay resort wanted its old promo pages out of Google, so the developer added Disallow: /promo/ to robots.txt and noindex to the pages. Google could no longer crawl them, so it never saw the noindex. The pages stayed in results as “Indexed, though blocked by robots.txt.” You will audit conflicts like this in Lesson 4.4.

Practice: crawling or indexing control?

Sort each item into the right group.

  • Disallow: /cart/ in robots.txt

  • <meta name="robots" content="noindex">

  • X-Robots-Tag: noindex on PDF files

  • Canonical tag pointing to the main version

  • Disallow: /search? in robots.txt

Common mistakes

  • Using robots.txt to deindex. It hides the noindex instead.
  • Assuming crawled means indexed. Check Search Console.
  • Trying to index everything. Some pages should be crawled but kept out of the index.
1. Which control stops Googlebot from fetching a URL?
2. Why does a noindex fail on a page blocked by robots.txt?
3. What does 'Crawled, currently not indexed' mean?

Your action

Next, you will audit robots.txt.

Key terms

Crawling
Googlebot requesting a URL and downloading its content.
Indexing
Google processing a page and storing it in its index so it can rank.
Indexed, though blocked by robots.txt
A Search Console status for URLs Google indexed from links alone, without reading the content, because robots.txt blocked crawling.
Crawled, currently not indexed
A status meaning Google fetched the page but chose not to index it, often due to quality or duplication.

Questions beginners ask

Can robots.txt remove a page from Google?

No. Robots.txt stops crawling, not indexing. To remove a page, allow crawling and use noindex, or remove the page.

Why would Google crawl a page and not index it?

Common reasons are duplicate content, thin content, low perceived value, or a canonical pointing elsewhere.

Does every crawled page need to be indexed?

No. Filter pages, internal search results, and thank-you pages are often better left out of the index.

Which comes first?

Discovery, then crawling, then rendering, then indexing, then ranking. A problem at any step stops the steps after it.

Related lessons

Want someone to handle SEO for your business while you learn? The team at Remar Digital can help.