Splitting Large Sitemaps With a Sitemap Index for Headless

Shard a sitemap that has outgrown the protocol limits into type- and locale-scoped children behind a sitemap index, so each publish only rebuilds the shard that actually changed.

When to Use This Approach

Move from a single sitemap to a sharded sitemap index when your catalog crosses the protocol ceilings:

• Your URL set exceeds 50,000 entries, the hard per-file limit, so a single sitemap file can no longer list every page.
• The uncompressed file approaches 50 MB, the other per-file ceiling, even before you hit the URL count.
• You run a per-locale or multi-type catalog where regenerating one giant file on every publish wastes compute and slows regeneration on CMS publish.

Implementation Steps

Step 1: Shard URLs by Type and Locale

Partition the URL set along stable boundaries — content type and locale — so each child stays comfortably under 50,000 URLs. Type and locale map directly onto CMS collections, which makes shard membership deterministic.

// lib/shard-urls.js — group URLs into named shards under the limit
const MAX_PER_SHARD = 45000; // headroom below the hard 50,000 ceiling

function shardUrls(urls) {
  const shards = new Map(); // key: `${type}-${locale}` -> URL[]
  for (const u of urls) {
    const base = `${u.type}-${u.locale}`;
    const bucket = shards.get(base) || [];
    // Overflow a huge collection into numbered sub-shards
    const idx = Math.floor(bucket.length / MAX_PER_SHARD);
    const key = idx > 0 ? `${base}-${idx + 1}` : base;
    const arr = shards.get(key) || [];
    arr.push(u);
    shards.set(key, arr);
    shards.set(base, bucket.concat(u));
  }
  return shards;
}

module.exports = { shardUrls, MAX_PER_SHARD };

Validation:

node -e "
const { shardUrls, MAX_PER_SHARD } = require('./lib/shard-urls');
const urls = require('./urls.json');
for (const [k, v] of shardUrls(urls)) if (v.length > MAX_PER_SHARD) console.log('OVER', k, v.length);
"
# No output means every shard is under the limit.

Step 2: Emit a Sitemap Index

Generate an index file that references each child shard with its own lastmod. The index is the one file you submit; crawlers follow it to the children.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://yourdomain.com/sitemaps/posts-en.xml.gz</loc>
    <lastmod>2026-07-06T09:12:00Z</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://yourdomain.com/sitemaps/posts-de.xml.gz</loc>
    <lastmod>2026-07-05T18:40:00Z</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://yourdomain.com/sitemaps/products.xml.gz</loc>
    <lastmod>2026-07-06T11:03:00Z</lastmod>
  </sitemap>
</sitemapindex>

Validation:

# The index must reference at most 50,000 children and parse cleanly
curl -s https://staging.yourdomain.com/sitemap-index.xml \
  | xmllint --noout - && echo "index OK"

Step 3: Gzip the Child Shards

Compress each child to gzip. The 50 MB limit applies to the uncompressed body, but gzip cuts transfer size dramatically and crawlers accept .xml.gz children natively.

// scripts/write-shards.js — serialize and gzip each shard
const { gzipSync } = require('node:zlib');
const { writeFileSync } = require('node:fs');

function writeShard(name, xml) {
  const raw = Buffer.from(xml, 'utf8');
  if (raw.byteLength > 50 * 1024 * 1024) {
    throw new Error(`${name} exceeds 50MB uncompressed — split further`);
  }
  writeFileSync(`public/sitemaps/${name}.xml.gz`, gzipSync(raw));
}

module.exports = { writeShard };

Validation:

# Confirm the decompressed shard is valid XML and under the ceiling
gzip -dc public/sitemaps/products.xml.gz | xmllint --noout - && \
  echo "uncompressed: $(gzip -dc public/sitemaps/products.xml.gz | wc -c) bytes"

Step 4: Regenerate Only Changed Shards

On publish, rebuild only the shard that owns the changed URL, then bump that child’s lastmod in the index. This is where sharding pays off — a single edit touches one small file, not the whole catalog. It pairs directly with regeneration on CMS publish.

// inside the publish webhook handler
const { shardKeyFor, rebuildShard, touchIndexEntry } = require('../lib/sitemap');

async function onPublish({ type, locale, path }) {
  const key = shardKeyFor(type, locale); // e.g. "posts-de"
  await rebuildShard(key);               // regenerate + gzip just this child
  await touchIndexEntry(key, new Date()); // update its <lastmod> in the index
  await purgeCdn([`/sitemaps/${key}.xml.gz`, '/sitemap-index.xml']);
}

Validation:

# After publishing a German post, only posts-de and the index should change
curl -s https://yourdomain.com/sitemap-index.xml \
  | grep -A1 "posts-de.xml.gz" | grep -oP '(?<=<lastmod>)[^<]+'
# Expected: a fresh timestamp; other children's lastmod must be unchanged.

Step 5: Validate the Index and Children in Search Console

Submit only the index URL and confirm every child is discovered and parsed. Keep the URLs inside each shard aligned with the canonical URL each page self-references.

# Fetch the index, list every child, and parse each one
for child in $(curl -s https://yourdomain.com/sitemap-index.xml \
  | grep -oP '(?<=<loc>)[^<]+'); do
  gzip -dc <(curl -s "$child") 2>/dev/null | xmllint --noout - \
    && echo "OK  $child" || echo "BAD $child"
done
# Then in GSC: submit only /sitemap-index.xml and watch child discovery.

SEO Impact Summary

Measurable signals to watch:

• GSC Sitemaps: each child shard should show “Success” with a discovered-URL count matching its shard size.
• Regeneration time per publish should fall sharply once only one shard rebuilds instead of the full set.
• No “too many URLs” or “sitemap file size” warnings should appear in the Sitemaps report after sharding.

Edge Cases and Gotchas

Empty shards after bulk unpublish Deleting an entire content type or locale can leave a child with zero URLs. An empty <urlset> is valid but pointless, so remove the child from the index and stop serving the file rather than shipping an empty shard that crawlers keep refetching.

Shard boundaries that drift over time If you shard by a numeric offset (“first 45k, next 45k”), a single insertion near the front cascades every URL into a different shard and forces a full regeneration. Shard by stable keys — type and locale — so a new entry lands in exactly one predictable child and never reshuffles the others.

Index-level versus URL-level lastmod confusion Crawlers use the lastmod on each <sitemap> reference in the index to decide which child to refetch. If you update per-URL lastmod inside a child but forget to bump the child’s entry in the index, the crawler has no reason to refetch that shard and the change goes unseen.

Locale and hreflang alignment When shards are split by locale, each locale’s URLs must still carry consistent signals with your international routing. A URL belongs in exactly one locale shard, and its canonical inside the sitemap must match the self-referencing canonical the page renders.

Frequently Asked Questions

What are the sitemap size limits? A single sitemap file may list at most 50,000 URLs and must not exceed 50 MB uncompressed. A sitemap index can itself reference up to 50,000 child sitemaps, which gives you enormous headroom once you shard. When you approach either per-file ceiling, split the URLs across children and list them in an index.

How should I shard a large sitemap? Shard along stable, meaningful boundaries such as content type and locale rather than arbitrary numeric chunks. Type and locale shards map cleanly to CMS collections, keep each child well under the URL limit, and let you regenerate only the shard that changed on publish. Avoid offset-based chunking, which reshuffles every URL when one is inserted.

Do sitemap children need their own lastmod? The most important lastmod is the one on each child reference inside the index file, because it tells crawlers which shard changed so they refetch only that one. Per-URL lastmod inside each child is still worthwhile for page-level recrawl signals, but keep the index-level timestamps accurate first — that is what drives efficient shard-level recrawl.

Can I mix gzipped and plain children in one index? Yes. Crawlers accept both .xml and .xml.gz children referenced from the same index, so you can compress only the large shards if you prefer. In practice gzipping all children is simplest and keeps transfer sizes uniformly small.

Part of: XML Sitemap Generation for Headless

Related

• Automating Sitemap Regeneration on CMS Publish — trigger per-shard regeneration from the CMS publish webhook
• Crawl Budget Impact in Headless — why accurate shard-level change signals conserve crawl budget on large sites
• Canonical URL Enforcement — keep every sharded URL aligned with the canonical its page self-references