Generating Sitemaps for Large Static Sites

On a 40-page site a sitemap is a formality: search engines find everything through links anyway. On a site with thousands of pages it becomes a working tool. It tells crawlers which URLs exist and which changed recently, it is how new sections get discovered quickly, and — through Search Console's per-sitemap reports — it is the best way to see which parts of the site are indexed and which are not. That only works if the sitemap is accurate: every indexable page listed, nothing else listed, and modification dates that mean something.

This guide generates sitemap indexes for Hugo, Astro and Eleventy sites with thousands of pages, split by section, with lastmod from Git, and adds a CI check that the sitemap and the build output agree. It is part of Technical SEO for Static Sites. The measurements come from a Hugo documentation site with 3,900 pages across API reference, guides, changelog and blog sections.

Prerequisites

A static site with a configured production origin (baseURL, site or equivalent).
Full Git history available at build time (fetch-depth: 0 in actions/checkout), if lastmod comes from Git.
A clear rule for which pages are indexable — for example, everything except pages with noindex: true, tag pages below a threshold, 404.html and search results.

What Goes in a Good Sitemap

The sitemap should list exactly the canonical, indexable, 200-status URLs of the site. Each exclusion matters:

Redirects — after a migration, generators often still emit stub pages for old paths. Listing them makes Search Console report "Page with redirect" for each.
noindex pages — listing a URL while telling crawlers not to index it is a contradiction; Search Console reports "Submitted URL marked noindex".
Non-canonical variants — only the URL that matches the page's own canonical tag.
Preview hosts — the sitemap must use the production origin, even when built in a preview environment, or better, not be submitted from previews at all.

On the documentation site the default plugin output contained 612 URLs that should not have been there, almost all redirect stubs and tag pages.

Hugo: Sections as Separate Sitemaps

Hugo generates a sitemap per language automatically, and a sitemap index when there are several languages. For one language it emits a single sitemap.xml. To split by section, override the sitemap template and add a sitemap output per section.

# hugo.toml
enableGitInfo = true

[sitemap]
  changefreq = ""
  priority = -1
  filename = "sitemap.xml"

[outputFormats.SectionSitemap]
  mediaType = "application/xml"
  baseName = "sitemap"
  isPlainText = false
  notAlternative = true

[outputs]
  section = ["HTML", "SectionSitemap"]

{{/* layouts/_default/section.sectionsitemap.xml */}}
{{ printf "<?xml version=\"1.0\" encoding=\"utf-8\"?>" | safeHTML }}
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
{{ range where .RegularPagesRecursive "Params.noindex" "!=" true }}
  <url>
    <loc>{{ .Permalink }}</loc>
    <lastmod>{{ .Lastmod.Format "2006-01-02T15:04:05Z07:00" }}</lastmod>
  </url>
{{ end }}
</urlset>

The root sitemap.xml then becomes an index listing /api/sitemap.xml, /guides/sitemap.xml and so on. With enableGitInfo, .Lastmod is the author date of the last commit touching the file, unless frontmatter overrides it.

Astro: The Sitemap Integration With a Filter

@astrojs/sitemap generates sitemap-index.xml and numbered sitemap files. Its filter option removes pages, and serialize sets lastmod per URL:

// astro.config.mjs
import sitemap from '@astrojs/sitemap';
import { execSync } from 'node:child_process';

const noindex = new Set(['/search/', '/404/']);
function gitDate(url) {
  const file = urlToSourceFile(url);                   // your mapping from URL to .md path
  if (!file) return undefined;
  const iso = execSync(`git log -1 --format=%cI -- "${file}"`).toString().trim();
  return iso || undefined;
}

export default defineConfig({
  site: 'https://docs.example.com',
  integrations: [sitemap({
    filter: (page) => !noindex.has(new URL(page).pathname) && !page.includes('/tags/'),
    serialize: (item) => ({ ...item, lastmod: gitDate(item.url) }),
    entryLimit: 10000
  })]
});

Calling git log per URL costs time on large sites: 3,900 calls added 38 s to the build. Precompute a map once with a single git log --name-only --format=%cI pass and look dates up from it; that took 1.9 s.

Measured with hyperfine on the same runner with full history checked out.

Eleventy: A Template Is Enough

Eleventy has no built-in sitemap, and a template is the simplest solution. Use pagination to write one sitemap per section and a small index:

---
permalink: /sitemaps/{{ section.key }}.xml
pagination:
  data: collections.sitemapSections
  size: 1
  alias: section
eleventyExcludeFromCollections: true
---
<?xml version="1.0" encoding="utf-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
{%- for item in section.pages %}
  <url><loc>{{ site.url }}{{ item.url }}</loc><lastmod>{{ item.data.gitLastmod | isoDate }}</lastmod></url>
{%- endfor %}
</urlset>

collections.sitemapSections groups collections.all by first path segment, filtering out pages with noindex or eleventyExcludeFromCollections. gitLastmod comes from a computed data value using the same one-pass Git map as above.

Submitting and Reading the Reports

Reference the index in robots.txt (Sitemap: https://docs.example.com/sitemap.xml) and submit it once in Search Console and Bing Webmaster Tools. After that, the per-sitemap indexing report is the useful part: on the documentation site it showed the API reference at 97% indexed and the changelog at 41%, which prompted merging hundreds of tiny changelog entries into monthly pages.

Pinging search engines after each deploy is no longer needed — Google retired its sitemap ping endpoint in 2023. Bing and Yandex support IndexNow for instant notification of changed URLs, which a deploy step can call with the list of URLs whose lastmod changed.

Per-section sitemaps turned one vague coverage number into a specific, fixable problem in one section.

The CI Check

The sitemap should agree with the build. A check parses every sitemap, collects the URLs, and compares them with the set of built HTML pages that are indexable — no noindex, canonical equals own URL, not a redirect stub:

// scripts/check-sitemap.mjs (core)
const listed = new Set(await readSitemapUrls('dist/sitemap.xml'));   // follows the index
const indexable = new Set(await indexablePagesFromHtml('dist'));
const missing = [...indexable].filter((u) => !listed.has(u));
const extra = [...listed].filter((u) => !indexable.has(u));
if (missing.length || extra.length) {
  console.error({ missing: missing.slice(0, 20), extra: extra.slice(0, 20) });
  process.exit(1);
}

It runs in about 3 s for 3,900 pages. It caught a template change that dropped the whole blog from the sitemap within the same pull request, which in Search Console would have shown up as a slow decline weeks later.

Images, Videos and News Extensions

The sitemap protocol has extensions for images, videos and news. They are optional, and for most static sites the image extension is the only one worth considering.

Image sitemaps list the images on each page with <image:image> entries. Search engines discover images from <img> elements anyway, so the extension mainly helps when images are loaded in ways crawlers may miss — CSS backgrounds, lazy loaders that set src from JavaScript, or galleries opened in a lightbox. If every content image is a plain <img> with a real src, skip it. If you do include it, generate it from the same data as the page's images and keep it to images you actually want in image search: screenshots and diagrams, not icons and decorative backgrounds.

Video sitemaps matter for sites that host their own video; a site embedding YouTube gets little from them because the video's canonical home is YouTube.

News sitemaps apply only to publishers accepted into Google News and list articles from the last 48 hours; a blog should not use them.

A final practical note on size: sitemaps may be served gzip-compressed (sitemap.xml.gz), but on a CDN that compresses XML responses automatically, plain .xml files are simpler and end up equally small on the wire. The 3,900-page documentation site's largest section sitemap was 412 KB uncompressed and 38 KB with brotli.

Measured Impact

Measure	Plugin defaults	Section sitemaps + Git lastmod + CI check
URLs in sitemap that should not be	612	0
Search Console "Submitted URL marked noindex"	188	0
Median days from edit to recrawl (60-page sample)	9	3
New section fully discovered after launch	16 days	5 days
Build time added	0 s	1.9 s

Pitfalls & Rollback

Shallow clones. With fetch-depth: 1, every file's last commit is the current one and every lastmod is today. Fetch full history or cache a date map.
Squash merges rewriting dates. Commit dates reflect the merge, which is usually what you want; author dates reflect the original edit. Pick one consistently.
Trailing slash mismatches. Sitemap URLs must match canonicals exactly.
Preview builds submitting sitemaps. Only production should reference the sitemap in robots.txt; see robots.txt and noindex for Preview and Staging.
Rollback: sitemaps are static files; restoring the previous template restores the previous output on the next deploy. Search engines refetch the index within days.

Conclusion

A large site's sitemap is a data product. Generate it from the same page list the build uses, split it by section so coverage problems have an address, take lastmod from real edits, and prove in CI that it matches the output. On the documentation site that removed 612 wrong URLs, cut recrawl delay by two thirds and exposed the one section that search engines were ignoring.

Parent: Technical SEO for Static Sites — the other build-time SEO artefacts.
Canonical URLs and Duplicate Content on Static Sites — sitemap URLs must equal canonicals.
SEO-Safe Pagination and Tag Pages — deciding which archive pages are indexable.
Generating Sitemaps and RSS in Next.js Static Export — the Next.js version.
hreflang and Language Alternates on Static Sites — alternates in sitemaps.

Filed underHow-tos