Comparing Rendered vs Source HTML for Headless Pages

The only reliable way to know what a crawler can read on the first pass is to fetch the page without a JavaScript engine and compare it with what the browser eventually produces.

When to Use This Approach

Run this diff when:

  • A page looks complete in the browser but is reported as thin or unindexed.
  • Structured data validates in a testing tool that renders, but rich results never appear.
  • You are migrating a template to a new rendering mode and need to prove nothing moved client-side.

Two Captures, One Question

The question is never “are they identical” — they never are. It is “does the critical set appear in both”.

Capturing both views and comparing only what matters A URL fetched twice — once without JavaScript and once in a headless browser — with both captures reduced to a critical element set before being diffed. one URL curl — no JS what the origin sent headless browser DOM after hydration extract critical set extract critical set diff must be empty Raw markup will always differ — only the extracted set has to match

Implementation Steps

Step 1: Capture the server response

curl -s \
  -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
  https://example.com/blog/my-post > source.html
wc -c source.html

Validation: Open the file. If the body is a single empty div, the route is client-rendered and the rest of the diff is a formality — you already have the answer.

Step 2: Capture the rendered DOM

// scripts/render.mjs
import { chromium } from 'playwright';
import { writeFileSync } from 'node:fs';

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto(process.argv[2], { waitUntil: 'networkidle' });
writeFileSync('rendered.html', await page.content());
await browser.close();

Validation: Compare byte counts. A rendered file several times larger than the source is normal; a source file that is a small fraction of the rendered one is the warning sign.

Step 3: Extract the critical set from both

// scripts/extract.mjs — same extraction applied to both captures
import { readFileSync } from 'node:fs';
import { JSDOM } from 'jsdom';

export function extract(path) {
  const { document } = new JSDOM(readFileSync(path, 'utf8')).window;
  const text = (sel) => document.querySelector(sel)?.textContent?.trim() ?? null;
  const attr = (sel, a) => document.querySelector(sel)?.getAttribute(a) ?? null;

  return {
    title: text('title'),
    description: attr('meta[name="description"]', 'content'),
    canonical: attr('link[rel="canonical"]', 'href'),
    robots: attr('meta[name="robots"]', 'content'),
    h1: [...document.querySelectorAll('h1')].map((e) => e.textContent.trim()),
    headings: [...document.querySelectorAll('h2, h3')].map((e) => e.textContent.trim()),
    internalLinks: [...document.querySelectorAll('a[href^="/"]')]
      .map((a) => a.getAttribute('href')).sort(),
    jsonLd: [...document.querySelectorAll('script[type="application/ld+json"]')]
      .map((s) => s.textContent.trim()).sort(),
    bodyWords: (document.querySelector('main')?.textContent ?? '').split(/\s+/).length,
  };
}

SEO impact: Reducing both captures to the same shape turns an unreadable markup diff into a short list of concrete defects.

Validation: Run the extraction on a known-good page and confirm every field is populated from the source capture.

Step 4: Diff and report

// scripts/diff.mjs
import { extract } from './extract.mjs';

const src = extract('source.html');
const rnd = extract('rendered.html');
const problems = [];

for (const key of ['title', 'description', 'canonical', 'robots']) {
  if (src[key] !== rnd[key]) problems.push({ key, source: src[key], rendered: rnd[key] });
}
if (src.h1.join('|') !== rnd.h1.join('|')) problems.push({ key: 'h1', source: src.h1, rendered: rnd.h1 });
if (src.jsonLd.length !== rnd.jsonLd.length) {
  problems.push({ key: 'jsonLd', source: src.jsonLd.length, rendered: rnd.jsonLd.length });
}
const missingLinks = rnd.internalLinks.filter((l) => !src.internalLinks.includes(l));
if (missingLinks.length) problems.push({ key: 'internalLinks', missing: missingLinks.slice(0, 20) });

const wordRatio = src.bodyWords / Math.max(rnd.bodyWords, 1);
if (wordRatio < 0.9) problems.push({ key: 'bodyWords', ratio: wordRatio.toFixed(2) });

console.log(JSON.stringify(problems, null, 2));
process.exit(problems.length ? 1 : 0);

Validation: An empty result means every SEO-critical element survives without JavaScript. Any entry is a specific, actionable defect rather than a vague suspicion.

Step 5: Gate the critical set in CI

# .github/workflows/render-parity.yml
- name: Render parity
  run: |
    for url in $(cat ci/representative-urls.txt); do
      curl -s "$url" > source.html
      node scripts/render.mjs "$url"
      node scripts/diff.mjs || exit 1
    done

Validation: Move a canonical into a client component on a branch and confirm the job fails. The gate is only meaningful once you have seen it fail.

Reading the Diff

Each field that differs points at a specific architectural cause, which shortens the fix considerably.

What each differing field tells you Five fields that can differ between source and rendered captures, each with the architectural cause it indicates and the corresponding remedy. field differs cause fix title or description head managed client-side move to the metadata API canonical injected after hydration emit server-side json-ld count script added by a component render in the server tree internal links list fetched on the client fetch during server render body word ratio main content client-rendered change the rendering mode The diff is a diagnosis, not just a warning

SEO Impact Summary

The cost of a client-only element depends on which pass discovers it, and the two passes are separated by a queue of unpredictable length.

First pass and render pass, separated by a queue The initial crawl reading server HTML, a render queue of variable length, and the rendering pass that finally sees client-injected elements. first pass server HTML only render queue hours to weeks render pass client elements finally seen Anything in the critical set that waits for the second pass is indexed late or not at all
Signal What improves What breaks if misconfigured
First-pass indexing Everything needed to index the page is available without rendering Elements added during hydration are discovered late or not at all
Structured data JSON-LD present in the source is read on the first pass Client-injected markup depends on the render queue succeeding
Link discovery Internal links in the source establish the crawl graph immediately Client-fetched navigation hides deep pages from the first pass
Diagnostic confidence A concrete list of client-only elements replaces speculation Without the diff, “Google renders JavaScript” ends the investigation prematurely

Measurable signals to watch:

  • Size of the critical-set diff per template, which should be zero.
  • Ratio of server-HTML body words to rendered body words, which should be close to one.
  • Count of internal links present only after rendering, which should be zero on content templates.

Edge Cases and Gotchas

waitUntil: 'networkidle' can hang on polling pages A page with a heartbeat request never reaches network idle. Use a fixed timeout plus an explicit wait for a selector that marks the content as ready.

Consent management can suppress content in the rendered capture If a consent script hides the main content until a choice is made, the rendered DOM may contain less than the source. Set the consent cookie in the headless context so you are comparing like with like.

The diff must run against production-equivalent output A development server ships different markup, different scripts, and often no minification. Run the comparison against a preview deployment built the way production is built.

Streaming responses need care when capturing the source A streamed response arrives in chunks, and curl will show the whole thing while a crawler may act on what it has. The behaviour is worth understanding separately — see streaming SSR and what Googlebot actually sees.

Frequently Asked Questions

Google renders JavaScript, so does this still matter? Yes. Rendering happens on a queued second pass that can lag the initial crawl, so anything only present after hydration is discovered later or not at all. It also assumes the render succeeds — one script error, one blocked resource, one timeout, and the server HTML is all that counts.

Which elements must exist in the server HTML? Title, description, canonical, robots directive, the single h1, the main body text, internal navigation links, and structured data. Anything determining whether the page is indexed, how it is described, or what is discovered from it belongs in the server response.

How different is too different? Element counts always differ, and that is harmless — hydration adds wrappers and controls. The measure is the critical set: if every element in it matches between captures, the page is safe however much other markup differs.

Should I run this on every URL? No — run it per template. Pages sharing a template share their rendering behaviour, so a representative URL per template catches everything a full crawl would, at a fraction of the cost. Add a URL to the set whenever a new template ships.

Part of: Debugging & Diagnostics for Headless Routing

Related