Robots & Crawl Directives in Headless Stacks

In a decoupled architecture at least four layers can independently tell Googlebot what to do with a URL — the CDN, edge middleware, the framework’s metadata API, and a CMS toggle an editor flipped last week. This section establishes a single source of truth for each directive, shows how to generate the artefacts from real route data, and gives the verification that proves the crawler receives what you intended rather than what some forgotten layer emitted.

Prerequisites

Before assigning directive ownership, confirm you have:

  • A complete route manifest — the list of routes your framework actually builds, not the list you think it builds. Generating it is covered in dynamic route generation.
  • Write access to edge middleware or CDN rules, since some directives can only be observed by inspecting headers you may not control.
  • A CMS field for index intent on every content type — a boolean or enum an editor can set, so directives are content decisions rather than deploy-time constants.
  • curl with header inspection, and ideally a crawler-user-agent fixture in your test suite.
  • A working canonical strategy. Robots directives and canonicals interact badly when both are used to solve the same duplication problem — settle canonicals first via canonical URL enforcement.

The Directive Layer Model

Each directive has exactly one layer where it belongs and several where it can do damage. robots.txt operates on the whole origin and is fetched once; it can only prevent a fetch. Meta robots lives in the HTML and can only be read if the fetch was allowed. X-Robots-Tag lives in the response headers and is the only option for responses that have no HTML at all.

Which layer owns which crawl directive Three concentric stages of a crawler request — origin gate, response headers, and rendered HTML — each labelled with the directive it can carry and what that directive can and cannot do. Googlebot request path — a directive can only act at its own stage robots.txt origin-wide, fetched once blocks the fetch only X-Robots-Tag per response header works for any file type meta robots inside rendered HTML needs the fetch allowed owner: build step owner: edge middleware owner: framework metadata generated from routes non-HTML responses only driven by a CMS field Blocking the fetch also blocks the noindex — the two directives are not interchangeable

The single most expensive mistake in this area follows directly from the diagram: disallowing a URL you also want de-indexed. The crawler never fetches the response, so it never reads the noindex, and the URL lingers in the index as a bare link indefinitely.

Step-by-Step Implementation Workflow

Step 1 — Inventory every layer that can emit a directive

Before you can own directives you have to find the ones already in play. Fetch a representative route and read every robots-relevant signal at once.

for path in / /blog/my-post /search?q=test /files/spec.pdf; do
  echo "== $path"
  curl -sD - -o /tmp/body "https://example.com$path" \
    | grep -iE '^(x-robots-tag|link):'
  grep -oE '<meta[^>]*name="robots"[^>]*>' /tmp/body
done

Validation: Every surprise in that output is an unowned directive. A X-Robots-Tag you did not expect on an HTML route is almost always left over from a preview or staging rule.

Step 2 — Assign one authoritative owner per directive

Write the ownership decision down in the repository so it survives staff turnover. A minimal policy file is enough.

# robots-policy.yml — the single source of truth for directive ownership
crawl_blocking:
  owner: build/robots-txt-generator
  scope: origin-wide path patterns only
index_suppression:
  owner: framework metadata layer
  source: cms field `seo.noindex`
non_html_directives:
  owner: edge middleware
  scope: [".pdf", ".json", ".csv", "/api/*"]
forbidden:
  - edge middleware setting X-Robots-Tag on text/html
  - CDN page rules adding noindex

Validation: Add a CI check that fails when middleware code sets an X-Robots-Tag on an HTML content type.

Step 3 — Generate robots.txt from the route manifest

A hand-maintained robots.txt drifts from reality within a release or two. Build it from the same data your router uses.

// scripts/build-robots.mjs
import { writeFileSync } from 'node:fs';
import { routes } from '../.build/route-manifest.json' with { type: 'json' };
import { cms } from '../lib/cms.mjs';

const settings = await cms.settings.get();
const blocked = routes
  .filter((r) => r.meta?.crawl === 'deny')
  .map((r) => r.pattern.replace(/\[\.\.\.[^\]]+\]/g, '*').replace(/\[[^\]]+\]/g, '*'));

const lines = [
  'User-agent: *',
  ...[...new Set(blocked)].sort().map((p) => `Disallow: ${p}`),
  'Disallow: /api/',
  '',
  `Sitemap: ${settings.siteUrl}/sitemap.xml`,
];
writeFileSync('public/robots.txt', lines.join('\n') + '\n');

Validation: curl -s https://example.com/robots.txt after deploy. Every Disallow must correspond to a route pattern that exists; a stale rule for a deleted route is a maintenance smell that eventually blocks something real.

Step 4 — Emit per-route directives from the framework

Index intent is a content decision, so it belongs to the metadata layer and should read a CMS field rather than a hard-coded list.

// app/blog/[slug]/page.jsx
export async function generateMetadata({ params }) {
  const post = await cms.posts.get(params.slug);
  return {
    title: post.title,
    robots: {
      index: !post.seo?.noindex,
      follow: true,
      googleBot: { index: !post.seo?.noindex, 'max-image-preview': 'large' },
    },
  };
}

SEO impact: Editors control index intent directly, so suppressing a thin taxonomy page no longer requires a deploy — and the value is visible in the CMS, where the person who made the decision can see it.

Validation: Toggle the field in the CMS, wait for revalidation, and confirm the served HTML flips between index and noindex.

Step 5 — Cover non-HTML responses at the edge

PDFs, JSON endpoints, and generated exports never pass through a template, so a header is the only mechanism available.

// middleware.js — headers for responses with no HTML to carry a meta tag
export const config = { matcher: ['/files/:path*', '/api/:path*', '/exports/:path*'] };

export function middleware(request) {
  const res = NextResponse.next();
  const path = new URL(request.url).pathname;
  if (path.startsWith('/api/') || path.startsWith('/exports/')) {
    res.headers.set('X-Robots-Tag', 'noindex, nofollow');
  } else if (path.endsWith('.pdf')) {
    res.headers.set('X-Robots-Tag', 'noindex, noarchive');
  }
  return res;
}

SEO impact: Stops machine-readable endpoints and internal exports from competing with real pages in the index, without adding a Disallow that would also hide them from your own diagnostics.

Validation: curl -sI https://example.com/exports/report.csv | grep -i x-robots-tag returns the directive; the same command against an HTML route returns nothing.

Directive Combinations That Behave Unexpectedly

Directives compose, and the composition rules are not symmetrical. The matrix below records the outcome for the pairs that appear most often in decoupled stacks.

Outcome matrix for combined crawl directives A two-by-three grid crossing robots.txt allow or disallow with page-level index, noindex, and no directive, showing the resulting crawler behaviour in each cell. page: index page: noindex page: no directive robots.txt Allow crawled indexed crawled dropped from index crawled indexed by default robots.txt Disallow not crawled may appear as bare URL noindex never read stays indexed — the trap not crawled may appear as bare URL Combined outcome for each pair of directives To remove a URL: allow the crawl, serve noindex, and keep it crawlable until it drops

HTTP Headers & CDN Directives Reference

Header / directive Value Rationale
X-Robots-Tag noindex, nofollow on /api/* JSON endpoints have no HTML to carry a meta tag and should never compete for index slots
X-Robots-Tag noindex, noarchive on generated files Keeps exports and reports out of results without hiding them from internal crawls
X-Robots-Tag absent on text/html HTML routes must get their directive from the metadata layer so it is testable in the template
robots.txt Disallow path patterns only, never a page you want de-indexed Blocking the fetch also blocks the noindex the crawler needs to read
robots.txt Sitemap absolute URL of the sitemap index Gives crawlers a discovery entry point independent of internal linking
Cache-Control on robots.txt public, max-age=300 Short enough that an emergency edit propagates, long enough to absorb crawler re-fetches

Validation Protocol

Directive ownership is only real if it is checkable, and each owner has a different verification command.

Verifying each directive owner Build-time robots generation, framework metadata, and edge middleware, each with the command that confirms its output. owner verified by build step fetch robots.txt and diff against routes framework metadata grep the served HTML for meta robots edge middleware inspect headers on non-HTML responses Three owners, three commands — none of them substitutes for another

Confirm the directive an HTML route actually serves

curl -s -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
  -D /tmp/h https://example.com/blog/my-post -o /tmp/b
grep -i 'x-robots-tag' /tmp/h            # expect: no output
grep -oE '<meta[^>]*name="robots"[^>]*>' /tmp/b

Confirm robots.txt parses and matches intent

curl -s https://example.com/robots.txt | tee /tmp/robots.txt
grep -c '^Disallow:' /tmp/robots.txt
grep -q '^Sitemap: https://' /tmp/robots.txt && echo "sitemap declared"

Assert no HTML route carries a noindex header in CI

# ci-robots-audit.sh — fails the build on a stray header directive
fail=0
while read -r url; do
  if curl -sI "$url" | grep -qi 'x-robots-tag:.*noindex'; then
    echo "unexpected noindex header: $url"; fail=1
  fi
done < indexable-urls.txt
exit $fail

Confirm a suppressed page is still crawlable

curl -sI https://example.com/tag/archive-2019 | head -n 1   # expect 200, not 403
curl -s https://example.com/tag/archive-2019 | grep -o 'noindex'

Troubleshooting

Symptom Root cause Fix
URL still indexed weeks after adding noindex The path is also disallowed, so the directive is never fetched Remove the Disallow, keep the noindex, wait for a recrawl
Entire section deindexed after a deploy Preview-environment middleware shipped to production Scope the middleware matcher by hostname, not path
Bare URLs with no snippet in results Disallowed but externally linked Allow the crawl and serve noindex so Google can drop it properly
Correct meta tag, page still suppressed An X-Robots-Tag header disagrees and wins Audit response headers; the most restrictive directive always applies
robots.txt blocks a route that no longer exists Hand-maintained file drifted from the route manifest Generate the file from build data as in Step 3
Directive correct in HTML, absent in Google’s rendered view Meta tag injected client-side after hydration Move it into server-rendered metadata — see injecting JSON-LD at the render layer for the same pattern applied to structured data

Directives are the last instrument, not the first

Crawl directives are the most reached-for tool in technical SEO and the one most often used in place of a structural fix. A page that should not be indexed is frequently a page that should not exist, and a path generating crawl waste is usually a path that something is linking to unnecessarily. Reaching for a directive resolves the symptom while leaving the generator in place, which is why sites accumulate long directive lists that nobody can explain.

The discipline worth adopting is to ask, before adding any directive, whether the URL needs to exist at all. If it does not, remove the thing that creates it. If it does but should not be linked, remove the link. Only when the URL genuinely needs to exist, needs to be reachable, and genuinely should not appear in results is a directive the right answer — and at that point it is a small, explicable list rather than an accumulated one.

This matters because directives have a maintenance cost that compounds. Each one is a rule somebody must understand before changing anything nearby, and a rule whose original justification is usually undocumented. A stack with four directives is auditable; a stack with forty is a hazard, and the difference is almost never that the second site is more complex.

Recording the reasoning, not just the rule

The single most useful habit in this area is writing down why each directive exists at the moment it is added. A Disallow with a comment naming the problem it solved and the date it was added can be evaluated later; the same line without a comment can only be left alone, because nobody can tell whether removing it is safe.

That record is what makes periodic review possible. Once a year, walk the list and ask whether each entry’s original justification still holds — routes get deleted, generators get fixed, and a surprising share of directives turn out to be protecting against something that no longer exists.

Pages in This Section

Frequently Asked Questions

Does robots.txt Disallow remove a URL from the index? No. Disallow prevents crawling, not indexing. A disallowed URL that is linked from elsewhere can still surface as a bare URL with no snippet, because Google never fetched the page to read a noindex. To remove a URL you must allow the crawl, serve noindex, and keep the path crawlable until it drops out.

Which wins when meta robots and X-Robots-Tag disagree? Google combines both sources and applies the most restrictive result, so a noindex in either place suppresses the page. That makes accidental conflicts silently destructive — a leftover staging header will override a perfectly correct meta tag, and the HTML gives no clue that anything is wrong.

Where should robots directives live in a headless stack? Index suppression belongs in the framework metadata layer driven by a CMS field, because that is where the decision originates and where it is testable. Crawl blocking belongs in a generated robots.txt. Edge middleware should only cover responses no template renders, such as file downloads and API endpoints.

How long does a noindex take to remove a page? It requires a recrawl, so the timescale is the crawl frequency of that URL — hours for a heavily linked page, weeks for a deep archive one. You can accelerate an urgent case with the Removals tool in Search Console, but that is a six-month suppression, not a removal: the noindex still has to be crawled for the change to be permanent.

Part of: Dynamic Routing & Indexation Workflows

Related