Setting Cache-Control Headers for Crawlable Pages

A crawlable page has two audiences with opposite needs — a browser that must not hold a stale copy and a shared cache that should hold one for as long as possible — and one header has to serve both.

When to Use This Approach

Set these deliberately when:

  • Time to first byte varies widely by region, which means most requests are missing the edge.
  • Published changes take unpredictable amounts of time to appear for visitors.
  • Crawler requests are reaching the origin rather than being served from cache.

Two Caches, Two Directives

max-age governs the visitor’s browser, which you cannot purge. s-maxage governs the shared cache, which you can. Conflating them forces you to pick a value that is wrong for one of them.

Where each cache directive applies Origin, shared edge cache, and browser cache in sequence, showing s-maxage governing the edge which can be purged, and max-age governing the browser which cannot. origin renders the page shared edge cache s-maxage — purgeable browser cache max-age — not purgeable long lifetime — this is where the performance comes from short lifetime You can invalidate the middle box; you cannot invalidate the right one So the two directives should almost never carry the same value

Implementation Steps

Step 1: Split the directives

// lib/cache-headers.js
export const CACHE = {
  // Browser: short, because it cannot be purged.
  // Shared: long, because a publish webhook purges it immediately.
  article:  'public, max-age=60, s-maxage=86400, stale-while-revalidate=604800',
  product:  'public, max-age=30, s-maxage=3600,  stale-while-revalidate=86400',
  listing:  'public, max-age=30, s-maxage=900,   stale-while-revalidate=3600',
  homepage: 'public, max-age=30, s-maxage=300,   stale-while-revalidate=3600',
  account:  'private, no-store',
};

SEO impact: Crawler requests are answered from the edge at edge latency, which raises the crawl rate Google will sustain on the property.

Validation: curl -sI <url> | grep -i cache-control shows both directives with different values.

Step 2: Group templates by volatility

// The band, not the template, decides the number
export const VOLATILITY = {
  minutes: 300,      // homepage, news index
  hours:   3600,     // product pages, category listings
  daily:   86400,    // articles, guides
  weekly:  604800,   // documentation, legal pages
};

export function sharedLifetime(template) {
  return VOLATILITY[TEMPLATE_BAND[template] ?? 'hours'];
}

Validation: Ask what the worst acceptable staleness is if the purge fails entirely. That answer is the band, and it is usually longer than people first say.

Step 3: Add stale-while-revalidate

This is the directive that removes the latency cost of expiry without extending staleness in practice.

Cache-Control: public, max-age=60, s-maxage=86400, stale-while-revalidate=604800

  0 ────────── 86400 ──────────────────── 691200 s
  fresh        stale but served instantly, refreshed in background

SEO impact: No visitor and no crawler ever waits for a revalidation render. The response after expiry is served immediately from cache while the refresh happens behind it.

Validation: Request a URL just past its s-maxage. The response should be fast and carry a stale cache indicator, and the following request should show the refreshed content.

Step 4: Keep personalised responses out of the shared cache

// middleware.js
export function middleware(request) {
  const res = NextResponse.next();
  const personalised =
    request.cookies.has('session') || request.cookies.has('__prerender_bypass');

  if (personalised) {
    res.headers.set('Cache-Control', 'private, no-store');
    res.headers.set('Vary', 'Cookie');
  }
  return res;
}

SEO impact: Prevents the worst caching failure — a personalised or draft response stored at the edge and served to a crawler, which can index another visitor’s view of the page.

Validation: Request with and without a session cookie and confirm the directives differ. The draft case is covered in securing draft mode routes in Next.js.

Step 5: Verify hit ratio and staleness

# Are crawler requests hitting the edge?
awk -F'\t' '$4 ~ /Googlebot/ { total++; if ($6 == "HIT" || $6 == "STALE") hits++ }
            END { printf "crawler cache hit ratio: %.1f%%\n", 100*hits/total }' \
  cdn-logs.tsv

# Does a publish appear within the promised window?
before=$(curl -s https://example.com/blog/my-post | md5sum)
# publish an edit
sleep 30
after=$(curl -s https://example.com/blog/my-post | md5sum)
[ "$before" != "$after" ] && echo "purge working"

Validation: A crawler hit ratio below about 80% means the cache key is fragmented — usually by a Vary header or a query parameter that should have been normalised.

Choosing the Numbers

Volatility bands make the choice mechanical, and the mechanical choice is almost always better than a per-template debate.

Volatility bands and the values each implies Four bands from minutes to weekly, each with a shared cache lifetime, a browser lifetime, and the templates that belong in it. band s-maxage max-age templates minutes 300 30 homepage, news index hours 3600 30 products, listings daily 86400 60 articles, guides weekly 604800 60 docs, legal The shared value assumes an explicit purge on publish backs it up Browser values stay short in every band, because they cannot be invalidated

SEO Impact Summary

The relationship between response speed and sustainable crawl rate is the reason these headers matter for SEO rather than only for visitors.

Response speed against sustained crawl rate A curve showing crawl rate rising as response time falls, with a throttling region where slow responses reduce the rate Google will sustain. sustained crawl rate origin response time edge-served throttling region
Signal What improves What breaks if misconfigured
Crawl rate Fast cached responses raise the request rate Google sustains Origin-served pages under crawl bursts produce timeouts and throttling
Time to first byte Edge-served HTML removes CMS latency from every measurement A fragmented cache key means most requests miss and hit the origin
Content accuracy A purge-backed long lifetime keeps content current and fast at once Relying on expiry alone leaves stale content live for the whole window
Privacy Personalised responses never enter a shared cache A cached session response served to a crawler can index private data

Measurable signals to watch:

  • Crawler cache hit ratio from CDN logs, targeted above 80%.
  • Regional time to first byte spread, which should narrow substantially.
  • Time from publish to visible change, which should be seconds rather than the cache lifetime.

Edge Cases and Gotchas

Vary fragments the cache key faster than anything else Vary: User-Agent multiplies your cache entries by the number of distinct agents, which effectively disables it. Vary only on Accept-Encoding and, where genuinely required, Cookie on private routes.

Query parameters usually create separate cache entries Every tracking parameter variant is a distinct key unless the CDN is configured to ignore them. Configure a parameter allowlist at the edge, matching the classification in canonicalising sorted and filtered listings.

no-cache does not mean “do not cache” It means “revalidate before serving”, which still stores the response. The directive that prevents storage is no-store, and confusing them is how personalised responses end up cached.

A 404 should be cacheable too, briefly Uncached 404s let a crawl of fabricated URLs hammer the origin. A short shared lifetime on error responses protects the origin without meaningfully delaying a page that starts existing.

Frequently Asked Questions

Does a long cache lifetime mean Google sees stale content? Only if you rely on expiry rather than invalidation. With a purge on publish the shared lifetime can be long, because a change removes the entry immediately. Expiry is the fallback for whatever the purge missed.

Should max-age and s-maxage be the same? Almost never. A browser cache you cannot purge should hold a page for minutes at most; a shared cache you can purge should hold it far longer, since that is where the benefit comes from.

Do crawlers respect Cache-Control? Google uses it as one input into recrawl scheduling but not as binding. The real effect is indirect and larger: cached responses are fast, and fast responses raise the crawl rate Google will sustain.

What about pages that must never be cached? Mark them private, no-store and keep them out of the sitemap. If a page cannot be cached it is almost always personalised, and personalised pages are rarely ones you want indexed anyway.

Part of: Edge Caching Behavior for SEO

Related