robots.txt and XML Sitemaps

robots.txt

A plain text file at the root of the site - https://example.edu/robots.txt - that tells crawlers which paths they may fetch.

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search
Allow: /

Sitemap: https://example.edu/sitemap.xml

What it does not do

This is the single most misunderstood point in technical SEO, so it is worth stating plainly.

robots.txt controls crawling, not indexing. A disallowed URL can still appear in search results.

If other pages link to a URL you have blocked, Google knows the URL exists. It cannot fetch the page, so it lists the URL with no description - often with a note saying no information is available.

Worse, because it never fetched the page, it never saw your noindex tag. Blocking a page you want removed from search actively prevents the removal.

Rules and traps

• It must be at the root. A file at /pages/robots.txt is ignored.
• Paths are case sensitive.
• The most specific matching rule wins.
• Never block CSS or JavaScript. Google needs them to render the page; blocking them can make a responsive site look broken to the crawler.
• It is public. Anyone can read it, so listing your admin paths there advertises them.

# block everything - a staging leftover that removes a whole site
User-agent: *
Disallow: /

That four line file, shipped to production, is one of the more expensive mistakes in web development. Check it at every launch.

XML sitemaps

A list of the URLs on your site that you want indexed, in a format crawlers read directly. It does not guarantee indexing, but it guarantees discovery.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.edu/</loc>
    <lastmod>2026-08-11</lastmod>
  </url>
  <url>
    <loc>https://example.edu/courses/design</loc>
    <lastmod>2026-07-30</lastmod>
  </url>
</urlset>

changefreq and priority appear in every sitemap tutorial and are ignored. Include loc, include lastmod if it is honest, and leave the rest out.

Limits and splitting

A sitemap may hold at most fifty thousand URLs and be at most fifty megabytes uncompressed. Beyond that, split it and add an index:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.edu/sitemap-pages.xml</loc>
    <lastmod>2026-08-11</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.edu/sitemap-articles.xml</loc>
  </sitemap>
</sitemapindex>

What belongs in it

Only URLs you want indexed:

• Return status 200, not a redirect or an error.
• Are not blocked in robots.txt.
• Do not carry noindex.
• Are canonical - the version you want indexed, not a duplicate.

A sitemap full of redirects and error pages is a signal that the site is not maintained, and Search Console reports every one of them.

Submitting it

1. Reference it from robots.txt with a Sitemap: line.
2. Submit it in Search Console, which then reports how many URLs were read and how many were indexed.

An image sitemap

<url>
  <loc>https://example.edu/courses/design</loc>
  <image:image>
    <image:loc>https://example.edu/images/studio.jpg</image:loc>
  </image:image>
</url>

Useful when images are loaded lazily or by script and might not be discovered from the markup alone.

Generating them

Both files should be generated, not hand written, on any site with more than a few pages. On a database driven site the sitemap is a query and a template:

<?php
header("Content-Type: application/xml; charset=utf-8");
echo "<?xml version="1.0" encoding="UTF-8"?>
";
echo "<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
";

foreach ($pages as $page) {
    printf(
        "  <url><loc>%s</loc><lastmod>%s</lastmod></url>
",
        htmlspecialchars($page["url"], ENT_XML1),
        $page["updated_at"]
    );
}

echo "</urlset>
";

A generated sitemap stays accurate. A hand written one is out of date within a month.

Important rules

• robots.txt must be at the root and is publicly readable.
• Disallow blocks crawling, not indexing.
• A blocked page never has its noindex read.
• Never block CSS or JavaScript.
• Sitemap URLs must be absolute and canonical.
• Fifty thousand URLs per sitemap.
• Neither file is a security measure.

Common mistakes

• Shipping Disallow: / from a staging environment.
• Blocking a page in robots.txt and expecting noindex to remove it.
• Blocking asset directories, so Google cannot render the page.
• Listing private paths in a public file.
• A sitemap containing redirects, 404s and non canonical URLs.
• Relative URLs in a sitemap.
• A lastmod that is always today.
• Never submitting the sitemap.

Best practices

• Generate both files from the same source of truth as the site.
• Reference the sitemap from robots.txt and submit it in Search Console.
• Only include canonical, indexable, status 200 URLs.
• Use lastmod honestly or omit it.
• Use noindex for exclusion and robots.txt for crawl budget.
• Check robots.txt in production immediately after every deployment.

Practice

1. Write a robots.txt blocking an admin area, allowing everything else, and referencing a sitemap.
2. Explain in two sentences why blocking a page prevents it being removed from search.
3. Generate a sitemap from a data source and validate the XML.
4. Check whether your production robots.txt blocks any CSS or JavaScript path.