freecourses.digital
Lesson progress: 0 of 3 activities done
All lessons in Technical SEO Auditing Lesson 2.1 of 84

Module 1: Introduction to Technical SEO Auditing

  1. 1.1What Is a Technical SEO Audit?
  2. 1.2Technical SEO Audit vs. General SEO Audit
  3. 1.3When Should You Audit a Website?
  4. 1.4Types of Technical SEO Issues
  5. 1.5Defining Audit Scope and Objectives

Module 2: Website Crawling and URL Discovery

  1. 2.1How SEO Crawlers Discover URLs
  2. 2.2Configuring a Website Crawl
  3. 2.3Crawl Scope, Limits, and Exclusions
  4. 2.4Internal Links and Crawl Depth
  5. 2.5Orphan Pages and URL Discovery
  6. 2.6Crawl Data vs. Search Engine Data
  7. 2.7Identifying Crawl Coverage Gaps

Module 3: HTTP Status Codes and Redirect Auditing

  1. 3.1Understanding HTTP Status Codes
  2. 3.2Finding 4xx Errors and Broken Links
  3. 3.3Identifying 5xx Server Errors
  4. 3.4Redirect Chains and Redirect Loops
  5. 3.5Internal Links to Redirected URLs
  6. 3.6301 vs. 302 Redirects
  7. 3.7Auditing 404 Pages and Soft 404s

Module 4: Crawlability and Indexability Auditing

  1. 4.1Crawling vs. Indexing
  2. 4.2Robots.txt Audit
  3. 4.3Meta Robots and X-Robots-Tag
  4. 4.4Noindex and Canonical Conflicts
  5. 4.5XML Sitemap Validation
  6. 4.6Canonical URL Auditing
  7. 4.7Identifying Indexability Conflicts
  8. 4.8Comparing Crawl Data with Google Search Console

Module 5: URL Structure and Duplicate Content Issues

  1. 5.1URL Structure and Normalization
  2. 5.2Duplicate URLs and Duplicate Pages
  3. 5.3HTTP vs. HTTPS and Hostname Variations
  4. 5.4Trailing Slashes and URL Parameters
  5. 5.5Pagination and Faceted Navigation
  6. 5.6Canonicalization Strategies
  7. 5.7Internal Linking to Preferred URLs

Module 6: Technical On-Page Auditing

  1. 6.1Missing and Duplicate Title Tags
  2. 6.2Missing and Duplicate Meta Descriptions
  3. 6.3Missing, Duplicate, and Multiple H1 Headings
  4. 6.4Heading Hierarchy and HTML Structure
  5. 6.5Missing Image Alt Attributes
  6. 6.6Internal Link and Anchor Text Issues
  7. 6.7Empty, Thin, and Near-Duplicate Pages
  8. 6.8Auditing HTML Metadata at Scale

Module 7: Website Performance and Core Web Vitals

  1. 7.1Core Web Vitals: LCP, INP, and CLS
  2. 7.2Lab Data vs. Field Data
  3. 7.3Identifying Slow Pages
  4. 7.4Image and Resource Optimization Opportunities
  5. 7.5JavaScript and CSS Performance
  6. 7.6Mobile Usability and Responsive Rendering
  7. 7.7Prioritizing Performance Fixes

Module 8: JavaScript SEO and Rendering

  1. 8.1How Search Engines Process JavaScript
  2. 8.2HTML Source vs. Rendered HTML
  3. 8.3Client-Side vs. Server-Side Rendering
  4. 8.4JavaScript-Generated Content and Links
  5. 8.5Rendering Failures and Missing Content
  6. 8.6Lazy Loading and Crawl Accessibility
  7. 8.7Diagnosing JavaScript SEO Problems

Module 9: Structured Data and Schema Auditing

  1. 9.1Structured Data and Schema.org Basics
  2. 9.2JSON-LD and Structured Data Formats
  3. 9.3Finding Structured Data Across a Website
  4. 9.4Detecting Invalid or Incomplete Markup
  5. 9.5Schema Validation and Rich Result Eligibility
  6. 9.6Conflicting or Duplicate Structured Data
  7. 9.7Prioritizing Schema Fixes

Module 10: Analyzing Audit Data and Prioritizing Issues

  1. 10.1Reading Crawl Reports and Exports
  2. 10.2Grouping URLs by Issue Type
  3. 10.3Identifying Root Causes vs. Symptoms
  4. 10.4Evaluating Severity, Impact, and Effort
  5. 10.5Separating Sitewide and Isolated Issues
  6. 10.6Identifying False Positives
  7. 10.7Building an Actionable Issue Register

Module 11: Technical SEO Audit Reporting

  1. 11.1Structuring a Technical SEO Audit Report
  2. 11.2Documenting Evidence and Affected URLs
  3. 11.3Writing Developer-Ready Recommendations
  4. 11.4Mapping Issues to Business and Search Impact
  5. 11.5Creating a Prioritized Remediation Roadmap
  6. 11.6Presenting Findings to Stakeholders
  7. 11.7Tracking Fixes and Ownership

Module 12: Fix Verification and Technical QA

  1. 12.1Re-Crawling After Fixes
  2. 12.2Comparing Before-and-After Crawl Results
  3. 12.3Verifying Redirects, Canonicals, and Indexability
  4. 12.4Detecting Regressions and Newly Introduced Issues
  5. 12.5Testing Website Changes Before Deployment
  6. 12.6Confirming Fixes with Search Console and Other Evidence
  7. 12.7Closing Audit Findings and Recording Results

How SEO Crawlers Discover URLs

Covered here

  • How a crawler starts, follows links, and builds its URL list
  • The role of XML sitemaps and robots.txt in discovery
  • Which kinds of links crawlers can and cannot follow
  • How SEO crawlers differ from Googlebot

Short answer

SEO crawlers discover URLs by starting from a seed page, usually the homepage, then following every internal link they find. They also read XML sitemaps listed in robots.txt or added manually. Pages with no links pointing to them stay invisible, which is why crawl results depend on your site's internal linking and sitemaps.

Picture a jeepney driver who only knows routes by following road signs. If a barangay has no sign pointing to it, the jeepney never goes there. SEO crawlers work the same way. They only reach pages that something points to.

How a crawl works, step by step

  1. Start at the seed URL. Usually the homepage, like https://www.example.ph/.
  2. Request the page and record its status code, headers, and HTML.
  3. Extract every link from the HTML: navigation, body text, footer, and pagination.
  4. Add new URLs to the crawl queue if they are in scope and not already seen.
  5. Repeat until the queue is empty or a limit is reached.

The result is a map of every URL reachable through links, with data for each one.

Other ways crawlers find URLs

  • XML sitemaps. Most crawlers can read your sitemap and add those URLs to the queue. Comparing “found by links” and “found only in sitemap” is how you spot orphan pages in Lesson 2.5.
  • Robots.txt. It can list sitemap locations, and its Disallow rules tell the crawler what to skip.
  • Uploaded lists. You can paste URLs from Search Console or analytics for list mode crawling.

Crawlers follow standard links: <a href="/rooms/deluxe/">Deluxe Room</a>. They usually cannot follow:

  • Buttons with onclick events and no href
  • Links that only appear after a user scrolls, clicks, or types
  • URLs inside forms or search boxes
  • Links in images without an anchor wrapper

SEO crawlers vs. Googlebot

Your crawler copies Googlebot’s behavior, but Google knows more. It sees backlinks from other websites, remembers URLs from years ago, and uses Search Console submissions. So Google may index pages your crawl never finds, and your crawl may find pages Google ignores. Lesson 2.6 covers how to compare the two.

Practice: can the crawler follow it?

Sort each item into the right group.

  • <a href="/menu/">Menu</a> in the header

  • A 'Load more' button with only an onclick event

  • A URL listed in the XML sitemap

  • A results page shown only after typing in a search box

  • Pagination link to /blog/page/2/

  • Page with zero internal links and not in the sitemap

Common mistakes

  • Assuming the crawl found everything. It found everything linked, which is not the same.
  • Crawling only the homepage seed on multi-domain sites. Add each subdomain as a seed if it is in scope.
  • Ignoring JavaScript links. If menus are built with scripts, enable rendering (Lesson 2.2).
1. Where does a crawl usually start?
2. Which link can a crawler follow?
3. Why might Google index a page your crawler missed?

Your action

Next, you will configure your first crawl so it behaves like Googlebot.

Key terms

Seed URL
The starting page of a crawl, usually the homepage.
Crawl queue
The list of discovered URLs waiting to be visited by the crawler.
Crawlable link
A standard HTML anchor with an href attribute, such as <a href="/rooms/">, which crawlers can follow.
User agent
The name a crawler sends to the server to identify itself, such as Googlebot Smartphone or Screaming Frog SEO Spider.

Questions beginners ask

Do SEO crawlers see the same pages as Google?

Not exactly. They follow similar rules, but Google also discovers URLs from external backlinks, past crawls, and Search Console submissions. Your crawler only sees what your site links to and what you give it.

Can crawlers follow links in JavaScript?

Only if JavaScript rendering is enabled and the link is a real <a href> element after rendering. Click handlers without an href are not followed by Google either.

Does a crawler obey robots.txt?

By default, yes. Most tools let you ignore robots.txt for auditing, but keep the default when you want to see what Google can reach.

Do nofollow links stop discovery?

Many crawlers skip nofollow links by default. Google treats nofollow as a hint, so it may still discover the URL another way.

Related lessons

Want someone to handle SEO for your business while you learn? The team at Remar Digital can help.