SEO Crawler

An SEO crawler is a tool you run on your own site to see it the way a search engine bot would. Screaming Frog SEO Spider and Sitebulb are the most common. The free version of Screaming Frog crawls up to 500 URLs per crawl, which covers most small business sites.

It works like Googlebot in one way: it starts from a URL and follows links, recording each page’s status code, title, meta description, headings, canonical tag, and outgoing links. You can set it to identify as Googlebot through the user-agent setting, and to obey or ignore robots.txt.

It differs from Googlebot in important ways. An SEO crawler only finds pages linked from where it started, while Google also discovers URLs through sitemaps, external links, and its history of your site. It has no idea what is in Google’s index or how pages rank; for that you need Search Console. And it renders JavaScript only if you switch rendering mode on, while Google renders most pages by default.

The output is a crawl export: a spreadsheet of every URL found and its data. That export is the raw material for a technical SEO audit, but the crawl itself is not the audit. Deciding which findings matter is the human part.