Diagnosing Indexation Gaps With GSC URL Inspection

Turn a vague “submitted and not indexed” count into a per-URL, machine-readable diagnosis by inspecting a sample through the GSC URL Inspection API and bucketing every result to a single fixable root cause.

When to Use This Approach

Reach for the URL Inspection API when:

• Search Console reports a growing pool of submitted-but-not-indexed URLs and you need to know why per URL, not just the aggregate count.
• You suspect a canonical dispute — pages that look correct but where Google may have chosen a different canonical than the one you declared.
• You are running post-launch or post-migration coverage checks and want a repeatable, scriptable audit rather than clicking through the Search Console UI one URL at a time.

The Inspection Diagnostic Flow

Implementation Steps

Step 1: Authenticate the URL Inspection API

Authorise a service account (or OAuth client) for the webmasters.readonly scope and add it as a user on the Search Console property.

# Confirm the credentials can read the property before scripting anything
export GOOGLE_APPLICATION_CREDENTIALS=./sc-service-account.json
node -e "
const { google } = require('googleapis');
(async () => {
  const auth = new google.auth.GoogleAuth({
    scopes: ['https://www.googleapis.com/auth/webmasters.readonly'],
  });
  const sc = google.searchconsole({ version: 'v1', auth });
  const { data } = await sc.sites.list();
  console.log(data.siteEntry.map(s => s.siteUrl).join('\n'));
})();
"

Validation: The command prints your property URL (for example https://example.com/). If it errors with a permission message, the service account is not yet added to the property.

Step 2: Batch-inspect a stratified sample

Inspect a sample that spans your template types so one dominant template does not hide problems in another.

// inspect.js — rate-limited batch inspection
import { google } from 'googleapis';
import { readFileSync } from 'node:fs';

const auth = new google.auth.GoogleAuth({
  scopes: ['https://www.googleapis.com/auth/webmasters.readonly'],
});
const sc = google.searchconsole({ version: 'v1', auth });
const SITE = 'https://example.com/';
const urls = readFileSync('sample.txt', 'utf8').trim().split('\n');

for (const url of urls) {
  const { data } = await sc.urlInspection.index.inspect({
    requestBody: { inspectionUrl: url, siteUrl: SITE },
  });
  const r = data.inspectionResult.indexStatusResult;
  console.log(JSON.stringify({
    url,
    verdict: r.verdict,
    coverageState: r.coverageState,
    userCanonical: r.userCanonical,
    googleCanonical: r.googleCanonical,
    lastCrawl: r.lastCrawlTime,
  }));
  await new Promise((res) => setTimeout(res, 1200));
}

Validation: node inspect.js > results.ndjson && wc -l results.ndjson — the line count equals your sample size, confirming every URL returned a result.

Step 3: Parse verdict, coverageState, and both canonicals

Flatten the results into a table you can scan and sort.

jq -r '[.url, .verdict, .coverageState,
        (if .userCanonical == .googleCanonical then "match" else "MISMATCH" end)]
       | @tsv' results.ndjson \
  | sort -k4 \
  | column -t -s $'\t'

Validation: Rows flagged MISMATCH are canonical disputes; rows with coverageState containing not indexed are your remaining work. The counts should reconcile with the Search Console coverage totals.

Step 4: Bucket each URL to one root cause

Assign a single cause per URL. This mapping is the whole point of the exercise — it converts diagnosis into an actionable work queue.

googleCanonical != userCanonical      -> canonical-dispute
coverageState "Crawled - not indexed" -> thin-or-duplicate
coverageState "Discovered - not..."   -> crawl-budget-starved
verdict "FAIL" / robots blocked       -> blocked
lastCrawl absent                       -> never-discovered

Validation: Every URL in results.ndjson maps to exactly one bucket; no URL is left unclassified. Related canonical fixes are covered in canonical URL enforcement.

Step 5: Fix, then re-inspect the same sample

Apply the bucket-specific remediation, then re-run Step 2 on the identical sample to prove movement.

node inspect.js > results-after.ndjson
diff <(jq -r '.url+" "+.coverageState' results.ndjson | sort) \
     <(jq -r '.url+" "+.coverageState' results-after.ndjson | sort)

Validation: The diff shows coverageState transitions toward Submitted and indexed. URLs that did not move need a different bucket assignment.

SEO Impact Summary

Measurable signals to watch:

• The MISMATCH (canonical dispute) count in your sample should trend to zero after enforcement fixes.
• Crawled - currently not indexed should convert to Submitted and indexed within 2–3 crawl cycles once content or canonical issues are resolved.
• The overall unindexed ratio from your sitemap diff should decline release over release.

Edge Cases and Gotchas

Inspection reflects the last crawl, not live HTML The API returns Google’s most recent understanding of the URL, which may predate your fix by days. Always note lastCrawlTime; if it is older than your deploy, request indexing or wait for a re-crawl before concluding the fix failed.

Quota forces sampling, not full audits The endpoint allows roughly 2,000 inspections per property per day. On a 200k-URL site you must sample and extrapolate. Stratify by template so the sample represents the whole, and pair it with CDN log parsing for population-level crawl behaviour the API cannot give you.

“Discovered - currently not indexed” is a budget signal This state means Google knows the URL exists but has not spent budget crawling it. It rarely responds to per-URL fixes; the lever is reducing crawl waste elsewhere so crawl budget reaches these URLs.

Property type changes the URL forms you must inspect A domain property spans http/https and every subdomain, while a URL-prefix property does not. Inspect the exact canonical form you serve, and confirm trailing-slash and protocol variants resolve to one URL before reading disputes as content problems.

Frequently Asked Questions

What does “Crawled - currently not indexed” mean? Google fetched the page but chose not to index it — the crawl itself succeeded, so this is not an access problem. The usual causes are thin or duplicate content, a soft duplicate of a stronger page, or a canonical the algorithm overrode. The fix is content or canonical enforcement work rather than crawl access.

How do I batch URL Inspection? The API inspects one URL per request, so batching is a rate-limited loop over your sample. The quota is roughly 2,000 inspections per property per day, so sample instead of inspecting everything, add a short delay between calls, and cache results so you do not re-spend quota on unchanged URLs.

What is googleCanonical vs userCanonical? userCanonical is the canonical you declared in the page; googleCanonical is the URL Google actually selected. When they differ, Google has overridden your declaration — the signature of a canonical dispute and a frequent reason a submitted URL stays unindexed.

Can I request indexing through the API? No. The URL Inspection API is read-only; it reports status but cannot submit a URL for indexing. Use the Search Console UI’s “Request indexing” for individual URLs, and rely on accurate, freshly-regenerated sitemaps for bulk re-discovery after fixes.

Part of: Debugging & Diagnostics for Headless Routing

Related

• Parsing CDN Logs for Googlebot Crawl Waste — the population-level crawl view that complements per-URL inspection
• Canonical URL Enforcement — resolve the canonical disputes that inspection surfaces as MISMATCH
• Crawl Budget Impact in Headless — address the budget starvation behind “Discovered - currently not indexed”