Comparing Before-and-After Crawl Results
Inside this lesson
- How crawl comparison works
- Which changes to look at first
- How to read added, removed, and changed URLs
- How to summarize the comparison for the report
Short answer
Comparing crawls means placing the baseline audit crawl beside the post-fix crawl to see what changed. Use your crawler's compare mode or match exports by URL, then review issue counts, added and removed URLs, and changed values like status codes, canonicals, and indexability. Improvements confirm fixes, while unexpected changes reveal regressions.
The clearest proof that an audit worked is a simple before-and-after table. The crawl comparison builds it for you.
How to compare
With compare mode: In Screaming Frog, switch to Compare mode, select the baseline and the new crawl, and run Crawl Analysis. Each tab shows Added, New, Removed, and Missing per issue.
Without compare mode: export both, match by URL, and add “changed” columns for status, indexability, canonical, title, and meta robots.
What to look at first
- Issue counts for every finding in your register.
- Indexability changes: pages newly indexable or newly non-indexable.
- Status code changes: new 4xx, 5xx, or redirects.
- Added URLs: unexpected new patterns.
- Removed URLs: pages that disappeared from the crawl, possibly orphaned.
Summarize it
| Finding | Before | After | Status |
|---|---|---|---|
| IDX-03 Noindex on location pages | 40 | 0 | Fixed |
| RED-02 Links to 301s in footer | 1,200 | 0 | Fixed |
| CAN-02 Canonical to HTTP | 48 | 6 | Partly fixed |
| NEW Non-200 canonicals on blog | 0 | 85 | New issue |
Practice: improvement or regression?
Sort each item into the right group.
-
Noindexed location pages: 40 to 0
Answer: Improvement.
-
New 404s: 0 to 230
Answer: Regression to investigate.
-
Redirect chains: 900 to 12
Answer: Improvement.
-
Indexable blog pages: 400 to 60
Answer: Regression to investigate.
-
Duplicate titles on products: 1,200 to 0
Answer: Improvement.
Common mistakes
- Looking only at fixed issues.
- Ignoring removed URLs.
- Comparing crawls with different settings.
Your action
Next, you will verify redirects, canonicals, and indexability in detail.
Key terms
- Baseline crawl
- The original audit crawl used as the reference point.
- Compare mode
- A crawler feature that shows differences between two crawls, such as Screaming Frog's Crawl Comparison.
- Added and removed URLs
- URLs present in only one of the two crawls.
- Changed URLs
- URLs in both crawls with different values, such as a new status code or canonical.
Questions beginners ask
Can I compare without compare mode?
Yes. Export both crawls and match by URL with XLOOKUP, then flag differences in key columns.
What if URL structures changed?
Use URL mapping in compare mode, or a redirect map, so old URLs line up with new ones.
Are added URLs always bad?
No. New content is expected. Watch for unexpected patterns, like thousands of new parameter URLs.
How do I present the comparison?
A short table of issue counts before and after, plus notes on any new issues.
Related lessons
Want someone to handle SEO for your business while you learn? The team at Remar Digital can help.