Crawling

Before Google can show your page, it has to fetch it. That fetch is the crawl. Many beginners think crawling, indexing, and ranking happen in one step. They are separate stages, and a problem can stop your page at any one of them.

Crawling vs indexing vs rendering

Think of a delivery rider in Quezon City. Crawling is the rider reaching your gate and picking up the package. Rendering is opening the box to see what is inside. Indexing is deciding whether the item goes on the store shelf.

In Google terms, Googlebot requests your URL and downloads the HTML. Rendering runs your JavaScript so Google sees the page the way a browser would. Indexing happens after that, when Google decides whether to store the page. A page can be crawled and still never be indexed.

What blocks or slows crawling

Several things stop Googlebot before it reaches your content:

• A robots.txt disallow rule tells Googlebot not to fetch a path.
• Links built without a real <a href> are hard for Google to follow, so link discovery suffers.
• Server errors (5xx) and slow server responses make Google crawl more carefully.
• A page with no internal links and no XML sitemap entry gives Google no path to find it.

One edge case surprises many site owners. A robots.txt block stops crawling, but it does not always stop indexing. Other sites may link to the blocked URL. Google can then list it using their anchor text.

Crawl budget is another common worry. For a small business site with 50 pages, it is rarely the problem. It matters mostly for very large or fast-changing sites.

How it shows up in real tools

You will meet crawling in Search Console. The Crawl Stats report shows how often Googlebot visits and which server responses it gets. URL Inspection shows the last crawl date for one page. Your server logs show the user-agent string, so you can see each Googlebot request directly.

Example

A Davao bakery adds a new page for ensaymada boxes priced at ₱450. Nobody links to it, and the sitemap is not updated. Googlebot has no route to the page, so it sits uncrawled for days. Adding one internal link from the menu page gives the crawler a path in.

For hands-on practice tracing how Google finds your pages, work through the linked lesson.