freecourses.digital
Lesson progress: 0 of 3 activities done
All lessons in Technical SEO Auditing Lesson 2.2 of 84

Module 1: Introduction to Technical SEO Auditing

  1. 1.1What Is a Technical SEO Audit?
  2. 1.2Technical SEO Audit vs. General SEO Audit
  3. 1.3When Should You Audit a Website?
  4. 1.4Types of Technical SEO Issues
  5. 1.5Defining Audit Scope and Objectives

Module 2: Website Crawling and URL Discovery

  1. 2.1How SEO Crawlers Discover URLs
  2. 2.2Configuring a Website Crawl
  3. 2.3Crawl Scope, Limits, and Exclusions
  4. 2.4Internal Links and Crawl Depth
  5. 2.5Orphan Pages and URL Discovery
  6. 2.6Crawl Data vs. Search Engine Data
  7. 2.7Identifying Crawl Coverage Gaps

Module 3: HTTP Status Codes and Redirect Auditing

  1. 3.1Understanding HTTP Status Codes
  2. 3.2Finding 4xx Errors and Broken Links
  3. 3.3Identifying 5xx Server Errors
  4. 3.4Redirect Chains and Redirect Loops
  5. 3.5Internal Links to Redirected URLs
  6. 3.6301 vs. 302 Redirects
  7. 3.7Auditing 404 Pages and Soft 404s

Module 4: Crawlability and Indexability Auditing

  1. 4.1Crawling vs. Indexing
  2. 4.2Robots.txt Audit
  3. 4.3Meta Robots and X-Robots-Tag
  4. 4.4Noindex and Canonical Conflicts
  5. 4.5XML Sitemap Validation
  6. 4.6Canonical URL Auditing
  7. 4.7Identifying Indexability Conflicts
  8. 4.8Comparing Crawl Data with Google Search Console

Module 5: URL Structure and Duplicate Content Issues

  1. 5.1URL Structure and Normalization
  2. 5.2Duplicate URLs and Duplicate Pages
  3. 5.3HTTP vs. HTTPS and Hostname Variations
  4. 5.4Trailing Slashes and URL Parameters
  5. 5.5Pagination and Faceted Navigation
  6. 5.6Canonicalization Strategies
  7. 5.7Internal Linking to Preferred URLs

Module 6: Technical On-Page Auditing

  1. 6.1Missing and Duplicate Title Tags
  2. 6.2Missing and Duplicate Meta Descriptions
  3. 6.3Missing, Duplicate, and Multiple H1 Headings
  4. 6.4Heading Hierarchy and HTML Structure
  5. 6.5Missing Image Alt Attributes
  6. 6.6Internal Link and Anchor Text Issues
  7. 6.7Empty, Thin, and Near-Duplicate Pages
  8. 6.8Auditing HTML Metadata at Scale

Module 7: Website Performance and Core Web Vitals

  1. 7.1Core Web Vitals: LCP, INP, and CLS
  2. 7.2Lab Data vs. Field Data
  3. 7.3Identifying Slow Pages
  4. 7.4Image and Resource Optimization Opportunities
  5. 7.5JavaScript and CSS Performance
  6. 7.6Mobile Usability and Responsive Rendering
  7. 7.7Prioritizing Performance Fixes

Module 8: JavaScript SEO and Rendering

  1. 8.1How Search Engines Process JavaScript
  2. 8.2HTML Source vs. Rendered HTML
  3. 8.3Client-Side vs. Server-Side Rendering
  4. 8.4JavaScript-Generated Content and Links
  5. 8.5Rendering Failures and Missing Content
  6. 8.6Lazy Loading and Crawl Accessibility
  7. 8.7Diagnosing JavaScript SEO Problems

Module 9: Structured Data and Schema Auditing

  1. 9.1Structured Data and Schema.org Basics
  2. 9.2JSON-LD and Structured Data Formats
  3. 9.3Finding Structured Data Across a Website
  4. 9.4Detecting Invalid or Incomplete Markup
  5. 9.5Schema Validation and Rich Result Eligibility
  6. 9.6Conflicting or Duplicate Structured Data
  7. 9.7Prioritizing Schema Fixes

Module 10: Analyzing Audit Data and Prioritizing Issues

  1. 10.1Reading Crawl Reports and Exports
  2. 10.2Grouping URLs by Issue Type
  3. 10.3Identifying Root Causes vs. Symptoms
  4. 10.4Evaluating Severity, Impact, and Effort
  5. 10.5Separating Sitewide and Isolated Issues
  6. 10.6Identifying False Positives
  7. 10.7Building an Actionable Issue Register

Module 11: Technical SEO Audit Reporting

  1. 11.1Structuring a Technical SEO Audit Report
  2. 11.2Documenting Evidence and Affected URLs
  3. 11.3Writing Developer-Ready Recommendations
  4. 11.4Mapping Issues to Business and Search Impact
  5. 11.5Creating a Prioritized Remediation Roadmap
  6. 11.6Presenting Findings to Stakeholders
  7. 11.7Tracking Fixes and Ownership

Module 12: Fix Verification and Technical QA

  1. 12.1Re-Crawling After Fixes
  2. 12.2Comparing Before-and-After Crawl Results
  3. 12.3Verifying Redirects, Canonicals, and Indexability
  4. 12.4Detecting Regressions and Newly Introduced Issues
  5. 12.5Testing Website Changes Before Deployment
  6. 12.6Confirming Fixes with Search Console and Other Evidence
  7. 12.7Closing Audit Findings and Recording Results

Configuring a Website Crawl

Your roadmap for this lesson

  • Which crawler settings matter before you press Start
  • Why to crawl as Googlebot Smartphone
  • When to turn on JavaScript rendering
  • How to set a safe crawl speed and save your configuration

Short answer

Configuring a website crawl means setting your crawler to behave like Googlebot before you start. Choose a smartphone user agent, enable JavaScript rendering if the site needs it, connect Google Search Console and sitemaps, set crawl speed so the server stays stable, and save the configuration so future crawls are comparable.

A crawl is only as good as its settings. Crawl a JavaScript site without rendering and you will report “no content” on every page. Crawl a small shared-hosting site at full speed and you may take it offline. Ten minutes of setup prevents both.

Walk through the setup

The menu names below follow Screaming Frog, but every major crawler has the same options.

Step 1 of 7

1. Set the user agent

Choose Googlebot Smartphone. Google uses mobile-first indexing, so you want to see what the mobile crawler sees. Some servers show different content to different user agents.

A quick test crawl

Before the full crawl, run 50 to 100 URLs. Check that:

  • Pages return 200, not 403 (blocked by a firewall).
  • Word counts look real, not near zero (rendering problem).
  • Titles and H1s are captured.

If a Bicol online pasalubong shop shows only 12 words per page in text-only mode but 600 with rendering, you know rendering is required.

Common mistakes

  • Desktop user agent by default. You may miss mobile-only problems.
  • Full speed on weak servers. You cause the 5xx errors you then report.
  • Forgetting API connections. You lose the traffic data that makes prioritization possible.
  • Changing settings between crawls. Comparisons stop meaning anything.
1. Which user agent should most audits use?
2. Your test crawl shows 10 words per page on a React site. What do you change?
3. During the crawl, many 503 errors appear. What should you do first?

Your action

Next, you will set crawl scope, limits, and exclusions so the crawl stays focused.

Key terms

Crawl configuration
The saved settings for a crawl, including user agent, rendering, speed, limits, and integrations.
JavaScript rendering
A crawl mode where the tool runs the page's scripts in a headless browser, like Google does, before reading the HTML.
Crawl speed
How many URLs per second the crawler requests. Too fast can slow or crash a small server.
API integration
Connecting the crawler to Search Console or analytics so each URL shows clicks, impressions, and indexing data.

Questions beginners ask

Should I always enable JavaScript rendering?

Turn it on for sites built with React, Vue, Angular, or heavy page builders, or when menus load by script. For simple HTML sites, text-only crawling is faster and enough.

What crawl speed is safe for a shared hosting site?

Start at 1 to 2 URLs per second. If the server returns 5xx errors or timeouts, slow down further and crawl at night.

Why save the crawl configuration?

So the next crawl uses identical settings. Different settings make before-and-after comparisons unreliable.

Should I tell the client before crawling?

Yes, especially for large or fragile sites. Firewalls may block your crawler, and the developer may need to allowlist your IP.

Related lessons

Want someone to handle SEO for your business while you learn? The team at Remar Digital can help.