freecourses.digital
Lesson progress: 0 of 2 activities done
All lessons in SEO for Beginners Lesson 4.4 of 26

Module 0: Introduction

  1. 0.1What is a search engine?
  2. 0.2How do search engines work?
  3. 0.3What is SEO?
  4. 0.4How to start SEO

Module 1: Keyword Research

  1. 1.1What are keywords in SEO?
  2. 1.2What is search intent?
  3. 1.3How to find keywords for your website
  4. 1.4Keyword difficulty
  5. 1.5Keyword mapping

Module 2: On-Page SEO

  1. 2.1What is on-page SEO?
  2. 2.2How to optimize a page for a keyword
  3. 2.3How to write title tags and meta descriptions
  4. 2.4How to use headings for SEO
  5. 2.5Internal linking for SEO

Module 3: Link Building

  1. 3.1What is link building?
  2. 3.2What makes a good backlink?
  3. 3.3How to get backlinks for beginners
  4. 3.4Blogger outreach

Module 4: Technical SEO Basics

  1. 4.1What is technical SEO?
  2. 4.2How to check if a page is indexed
  3. 4.3How to find broken links on a website
  4. 4.4XML sitemap and robots.txt for beginners
  5. 4.5How to do a basic SEO audit

Module 5: Measuring SEO Results

  1. 5.1How to set up Google Search Console
  2. 5.2Clicks, impressions, CTR, and position explained
  3. 5.3Your SEO action plan

XML sitemap and robots.txt for beginners

Inside this lesson

  • What an XML sitemap does
  • What robots.txt controls
  • How to submit a sitemap to Google and Bing

Short answer

An XML sitemap is a file that lists the pages you want search engines to find. A robots.txt file gives crawlers rules about which parts of your site they may visit. The sitemap says “please look here.” Robots.txt says “please do not go there.” Submit your sitemap in Google Search Console and Bing Webmaster Tools.

What an XML sitemap is

An XML sitemap is like a table of contents for search engines. It lists your important URLs, and can include when each page was last updated.

It usually lives at an address like:

  • yoursite.com/sitemap.xml
  • yoursite.com/sitemap_index.xml (common in WordPress SEO plugins)

Most website builders, like WordPress with Yoast or Rank Math, Shopify, Wix, and Squarespace, create a sitemap for you.

A sitemap helps search engines find pages. It does not guarantee they will index them.

What robots.txt is

Robots.txt is a small text file at yoursite.com/robots.txt. Crawlers read it before visiting your site.

A simple example:

User-agent: *
Disallow: /cart/
Disallow: /admin/

Sitemap: https://yoursite.com/sitemap.xml

What each line means:

  • User-agent: * These rules apply to all crawlers.
  • Disallow: /cart/ Do not crawl pages in the cart folder.
  • Sitemap: Here is where my sitemap is.

The most dangerous line is Disallow: / with nothing after the slash. It blocks your entire site.

XML sitemap invites crawlers to pages, robots.txt asks them to skip some pathssitemap.xml“Please look here”✓ /services/✓ /blog/seo-tips/✓ /contact/robots.txt“Please do not go here”✗ /cart/✗ /admin/Never: Disallow: /
The sitemap invites. Robots.txt sets limits.

Robots.txt blocks crawling, not indexing

This surprises many beginners. Robots.txt stops crawlers from reading a page. But a blocked page can still appear in results if other sites link to it, usually without a description.

To keep a page out of search results, use a noindex tag instead, and do not block it in robots.txt. Crawlers must be able to read the page to see the noindex tag.

How to submit your sitemap

  1. Find your sitemap URL. Try yoursite.com/sitemap.xml in your browser.
  2. Google Search Console: Open Indexing, then Sitemaps. Paste the sitemap URL and click Submit.
  3. Bing Webmaster Tools: Open Sitemaps, click Submit sitemap, and paste the URL.
  4. Check the status after a few days. “Success” means the search engine read it.

Allowed or blocked quiz

Each question shows a robots.txt rule and a URL. The reader picks Allowed or Blocked.

1. Rule: Disallow: /admin/ URL: /admin/login
2. Rule: Disallow: /admin/ URL: /blog/seo-tips
3. Rule: Disallow: / URL: /products/
4. Rule: Disallow: (empty) URL: /contact/

Common mistakes

  • Leaving Disallow: / on after launching a new site. Check robots.txt on launch day.
  • Using robots.txt to hide private pages. Anyone can read robots.txt. Use passwords for private content.
  • Listing broken or redirected URLs in the sitemap. Keep it to working, indexable pages.
  • Never submitting the sitemap. Submit it to both Google and Bing.

Your action

Key terms

XML sitemap
A file listing pages you want search engines to find.
robots.txt
A file with rules telling crawlers which parts of a site they may visit.
User-agent
The name of a crawler that a robots.txt rule applies to.
Disallow
A robots.txt rule that blocks crawling of a path.
Noindex tag
A tag that keeps a page out of search results.

Questions beginners ask

Does a small site need a sitemap?

It helps, and most builders create one automatically. It matters more for new sites with few links.

Can robots.txt block AI crawlers?

Many AI crawlers follow robots.txt rules, like OpenAI’s GPTBot. Course 7 covers whether to allow them.

Want someone to handle SEO for your business while you learn? The team at Remar Digital can help.