Debugging & Diagnostics for Headless Routing

When a headless build ships hundreds of dynamically generated routes, “the pages aren’t indexed” is a symptom with a dozen possible causes spread across the CMS, the framework, the edge, and Google’s own pipeline. This section is a repeatable diagnostic protocol that isolates where a URL falls out of the crawl-to-index funnel so you fix the actual failure instead of guessing.

Prerequisites

Before running the diagnostics below, confirm you have:

• A verified Search Console property with API access enabled (a service account or OAuth client authorised for the URL Inspection API).
• CDN or origin access logs with request path, status code, user agent, client IP, and response time — Cloudflare Logpush, Fastly, CloudFront, or an origin log stream.
• curl, jq, and standard GNU coreutils (sort, uniq, awk) available in your shell and CI runner.
• The site’s canonical sitemap URL and, ideally, a scheduled export of indexed URLs from the Search Console API.
• A baseline crawl budget picture — see crawl budget impact in headless deployments for how request volume maps to index coverage.

If canonical URL enforcement is not already consistent across layers, expect canonical disputes to dominate your diagnostic results — resolve those first, because they mask other signals.

The Crawl-to-Index Funnel

Every indexed URL passes four gates: it must be discovered (submitted), fetched (crawled), rendered into HTML Google can read, and finally selected for the index. A different tool answers each gate, and the whole diagnostic is a matter of finding the first gate a URL fails.

Step-by-Step Implementation Workflow

Step 1 — Diff submitted URLs against indexed URLs

Start by quantifying the gap. Pull every URL from your sitemap and compare it to the set Google reports as indexed. The difference is your work queue.

# Extract loc entries from the sitemap
curl -s https://example.com/sitemap.xml \
  | grep -oP '(?<=<loc>)[^<]+' \
  | sort -u > sitemap-urls.txt

# indexed-urls.txt is exported from the Search Console API (see Step 2 tooling)
comm -23 sitemap-urls.txt indexed-urls.txt > unindexed-urls.txt
wc -l unindexed-urls.txt

The unindexed-urls.txt file is the population you diagnose. If it is a large fraction of the sitemap, you likely have a systemic routing or rendering problem rather than a per-page one.

Step 2 — Inspect a representative sample in GSC

Do not inspect thousands of URLs by hand. Sample the unindexed set (stratified by template type — article, listing, taxonomy) and run each through the URL Inspection API to read the machine-readable verdict.

# Sample 50 URLs across templates, then inspect (tooling shown in the next section)
shuf unindexed-urls.txt | head -n 50 > sample.txt
node inspect-batch.js sample.txt > inspection.json
jq -r '.[] | [.url, .verdict, .coverageState] | @tsv' inspection.json

The full inspection workflow lives in diagnosing indexation gaps with GSC URL Inspection.

Step 3 — Filter CDN logs for verified Googlebot

The inspection tells you Google’s conclusion; the logs tell you what actually happened at your edge. Filter to verified Googlebot so you are not misled by spoofed user agents.

# Keep only Googlebot user-agent lines, then verify by reverse DNS in the next section
zcat cdn-logs-*.gz \
  | awk -F'\t' '$6 ~ /Googlebot/ { print $1"\t"$4"\t"$5 }' \
  > googlebot-hits.tsv
wc -l googlebot-hits.tsv

Step 4 — Correlate status codes and response time per path

Aggregate crawler hits by path and status to see where budget is being spent and which paths return errors or slow responses that suppress indexing.

# Count Googlebot hits by status code
cut -f2,3 googlebot-hits.tsv | sort | uniq -c | sort -rn | head -n 20

A wall of 301, 302, or 404 responses on crawled paths is crawl waste — trace it with redirect chain management. The complete log-parsing method is in parsing CDN logs for Googlebot crawl waste.

Step 5 — Classify each gap to one root cause

Assign every unindexed URL to exactly one bucket. The bucket determines the fix owner and the remediation.

not-crawled       -> discovery/crawl-budget problem (internal links, sitemap freshness)
crawled-blocked   -> robots.txt / noindex / X-Robots-Tag suppressing the URL
canonical-dispute -> Google chose a different canonical (fix enforcement)
thin-render       -> server HTML lacks main content (rendering problem)
error-status      -> 4xx/5xx or redirect hop returned to the crawler

Diagnostic Tooling Examples

GSC URL Inspection batch client

The URL Inspection API accepts one URL per call, so a batch client is a rate-limited loop. Read verdict, coverageState, and the two canonicals.

// inspect-batch.js
import { google } from 'googleapis';
import { readFileSync } from 'node:fs';

const auth = new google.auth.GoogleAuth({
  scopes: ['https://www.googleapis.com/auth/webmasters.readonly'],
});
const search = google.searchconsole({ version: 'v1', auth });
const SITE = 'https://example.com/';

const urls = readFileSync(process.argv[2], 'utf8').trim().split('\n');
const out = [];
for (const url of urls) {
  const { data } = await search.urlInspection.index.inspect({
    requestBody: { inspectionUrl: url, siteUrl: SITE },
  });
  const r = data.inspectionResult.indexStatusResult;
  out.push({
    url,
    verdict: r.verdict,
    coverageState: r.coverageState,
    userCanonical: r.userCanonical,
    googleCanonical: r.googleCanonical,
  });
  await new Promise((r) => setTimeout(r, 1200)); // stay under quota
}
process.stdout.write(JSON.stringify(out, null, 2));

SEO impact: Turns an opaque “not indexed” status into a specific machine-readable reason per URL, so remediation targets the real gate instead of applying blanket fixes that do not move coverage.

Validation: Re-run the client on the same sample after a fix and confirm coverageState transitions toward Submitted and indexed.

Verified Googlebot log filter

Filter by user agent first, then confirm each source IP resolves to a googlebot.com / google.com host and forward-resolves back — the only reliable way to exclude spoofed crawlers.

# verify-googlebot.sh — reverse+forward DNS confirmation
while read -r ip; do
  host=$(dig +short -x "$ip" | sed 's/\.$//')
  case "$host" in
    *.googlebot.com|*.google.com)
      fwd=$(dig +short "$host" | tr '\n' ' ')
      [[ " $fwd " == *" $ip "* ]] && echo "$ip verified"
      ;;
  esac
done < <(cut -f1 googlebot-hits.tsv | sort -u)

SEO impact: Ensures crawl-waste conclusions are drawn from real Googlebot traffic, preventing wasted engineering time chasing bot-imposter noise.

Validation: Spot-check a “verified” IP against Google’s published crawler ranges and confirm it falls inside them.

Sitemap-to-index diff in CI

Automate Step 1 so the unindexed ratio becomes a tracked metric, not a manual chore.

# ci-index-diff.sh — fails the job when the unindexed ratio is too high
total=$(wc -l < sitemap-urls.txt)
missing=$(comm -23 sitemap-urls.txt indexed-urls.txt | wc -l)
ratio=$(awk -v m="$missing" -v t="$total" 'BEGIN { printf "%.3f", m/t }')
echo "unindexed ratio: $ratio ($missing / $total)"
awk -v r="$ratio" 'BEGIN { exit (r > 0.15) }' \
  || { echo "unindexed ratio above 15% threshold"; exit 1; }

SEO impact: Catches indexation regressions within a crawl cycle instead of when traffic drops weeks later.

Validation: Introduce a deliberately noindexed test URL into the sitemap and confirm the job flags the increased ratio.

HTTP Headers & CDN Directives Reference

Validation Protocol

Confirm a URL’s live status and canonical

curl -sD - -o /dev/null https://example.com/blog/my-post \
  | grep -iE 'http/|x-robots-tag|link'
# Expected: HTTP/2 200, no noindex X-Robots-Tag, Link canonical == page URL

Confirm server HTML contains main content (no CSR-only shell)

# Byte count of server HTML text content — a near-empty shell renders thin
curl -s https://example.com/blog/my-post \
  | grep -oP '(?<=<main).*?(?=</main>)' | wc -c

Re-inspect after a fix

node inspect-batch.js sample.txt \
  | jq -r '.[] | select(.coverageState != "Submitted and indexed") | .url'
# The list should shrink after each remediation cycle

Troubleshooting

Pages in This Section

• Diagnosing Indexation Gaps With GSC URL Inspection — batch-inspect URLs through the API and classify each verdict and coverage state to a root cause
• Parsing CDN Logs for Googlebot Crawl Waste — verify real Googlebot, aggregate hits by path and status, and quantify wasted crawl budget

Frequently Asked Questions

How do I tell if a URL is crawled but not indexed? Run the URL through GSC URL Inspection. A coverageState of “Crawled - currently not indexed” with a recent last-crawl date confirms Googlebot fetched the page but declined to index it — which usually points to thin content, a canonical dispute, or a soft duplicate rather than a crawl-access problem. Compare the Google-selected canonical against your declared one before assuming the content is at fault.

What CDN log fields reveal crawl waste? You need the request path, HTTP status, user agent, client IP for Googlebot verification, and response time. Aggregating hits by path and status exposes budget burned on parameterised URLs, redirect hops, and 404s that never produce an indexable page.

How often should I run a sitemap-to-index diff? Weekly for sites publishing daily, and after every migration or large release. The diff automates cheaply in CI, so schedule it and alert when the unindexed ratio crosses a threshold rather than checking by hand.

Does a 5xx spike hurt indexing beyond the affected URLs? Yes. Sustained 5xx or 503 responses signal instability and Google throttles crawl rate across the whole property to avoid overloading it, which shrinks the effective crawl budget available to healthy URLs. Treat error-rate spikes as a site-wide indexation risk, not a per-URL nuisance.

Part of: Dynamic Routing & Indexation Workflows

Related

• Canonical URL Enforcement — resolve the canonical disputes that dominate “crawled, not indexed” diagnoses
• Redirect Chain Management — eliminate the redirect hops that show up as crawl waste in log analysis
• Crawl Budget Impact in Headless — understand how wasted requests reduce the budget available for indexable pages
• XML Sitemap Generation for Headless — keep the submitted URL set accurate so the index diff reflects real gaps
• Slug Normalization Strategies — remove slug variants that fragment indexing and muddy diagnostics