Back to Blog

SEO

robots.txt and sitemap.xml: what they do and what to watch for

Admin User
robots.txt and sitemap.xml: what they do and what to watch for

These two files are constantly mentioned together in SEO and constantly confused. One tells bots "don't go there," the other says "these are the important ones." Neither guarantees anything.

robots.txt: crawl control, not privacy

robots.txt is a plain-text file at the root of your domain that tells crawlers which paths not to crawl. The trap everyone falls into: per Google's documentation, robots.txt is not a way to hide a page from Google. If other sites link to a URL that's blocked from crawling, Google may still index the URL without seeing its content. If you want a page to stay out of results entirely, use noindex or password-protect it.

There's a reverse trap too: if you put noindex on a page and also block it in robots.txt, Google can never see the noindex tag. Don't combine the two.

A typical file

User-agent: *
Disallow: /admin/
Disallow: /api/
Sitemap: https://www.example.com/sitemap.xml

Common mistakes

  • A Disallow: / line carried from staging to production. The whole site becomes uncrawlable; always check before launch.
  • Blocking CSS and JavaScript files; Google can't render the page properly.
  • Treating robots.txt as a security measure. The file is public; listing secret paths there gives them away.

sitemap.xml: a discovery aid

A sitemap is a file that tells Google which URLs on your site matter. According to Google, a sitemap isn't required for every site: small sites of roughly 500 pages or fewer that are well linked usually don't need one. For large or new sites, sites whose pages are poorly linked, and media-heavy sites, it can speed up discovery. But having a URL in the sitemap doesn't guarantee it'll be indexed.

  • Include only URLs you want indexed that return 200 and are canonical. Redirected, noindex or error pages pollute the sitemap.
  • On multilingual sites you can include hreflang alternates in the sitemap too.
  • Submit it under Search Console > Sitemaps and reference it in robots.txt with a Sitemap: line.
  • Update "lastmod" only when the content has actually changed.

Checklist

  1. Open /robots.txt and /sitemap.xml in a browser; do they return 200?
  2. In Search Console, confirm the sitemap shows "Success" and the discovered URL count matches expectations.
  3. If an important page isn't indexed, use URL Inspection to find out why; the fix is more often content and internal links than robots.txt. See the technical SEO audit.

Sources