Serving X-Robots-Tag From Edge Middleware

Header directives are the only way to control indexing for responses that never pass through a template — and the fastest way to deindex an entire site if the matcher is one character too broad.

When to Use This Approach

Use a header directive when:

  • The response is not HTML — a JSON endpoint, a CSV export, a generated PDF, or a file download — so there is no <head> to carry a meta tag.
  • You need a directive on a response your framework does not render, such as a static asset served directly by the CDN.
  • An entire non-production hostname must be suppressed, where a per-route approach would miss the endpoints nobody remembers.

For ordinary HTML pages, the directive belongs in the framework metadata layer instead — the reasoning is in robots & crawl directives in headless stacks.

Why Scope Is the Whole Problem

A header directive applies to whatever the matcher catches, and matchers are written once and then forgotten. The diagram contrasts a matcher scoped by response type against one scoped by a path prefix that later grew.

Matcher scope decides whether a header directive is safe Two request flows: a narrow matcher passing HTML routes through untouched while tagging JSON and file responses, and a broad matcher that tags an HTML route added under the same prefix. scoped by content type — safe scoped by path prefix — fragile /api/products /files/spec.pdf /blog/my-post is it text/html? no → tag it page passes through clean /exports/report.csv /exports/2026/ match /exports/* tag everything the new landing page is silently deindexed A path prefix describes today's routes; a content type describes the response itself Match on what the response is, not on where it happens to live

Implementation Steps

Step 1: Enumerate the responses that have no HTML

# Every route the framework builds that is not a page
find app -name 'route.js' -o -name 'route.ts' | sed 's|app||; s|/route\..*||'
# Plus anything served straight from the CDN
aws s3 ls s3://example-assets/exports/ --recursive | awk '{print $4}' | head

Validation: The list should contain only machine-readable endpoints and downloads. If a page route appears, the enumeration is wrong, not the middleware.

Step 2: Decide the directive from the content type

Matching on the response rather than the request is what makes the rule survive a routing change.

// middleware.js
import { NextResponse } from 'next/server';

const DIRECTIVES = [
  { test: (ct) => ct.includes('application/json'), value: 'noindex, nofollow' },
  { test: (ct) => ct.includes('text/csv'), value: 'noindex, noarchive' },
  { test: (ct) => ct.includes('application/pdf'), value: 'noindex, noarchive' },
];

export function middleware(request) {
  const res = NextResponse.next();
  const ct = res.headers.get('content-type') ?? '';
  if (ct.includes('text/html')) return res;   // pages are never tagged here
  const rule = DIRECTIVES.find((d) => d.test(ct));
  if (rule) res.headers.set('X-Robots-Tag', rule.value);
  return res;
}

SEO impact: A route that changes from returning JSON to returning a rendered page loses the directive automatically, which is the safe direction for the change to fail in.

Validation: curl -sI https://example.com/api/products | grep -i x-robots-tag returns the directive; the same request to /blog/my-post returns nothing.

Step 3: Restrict the matcher so the middleware barely runs

Content-type checking is the safety net, not the primary control. The matcher should keep the middleware off page routes entirely.

export const config = {
  matcher: [
    '/api/:path*',
    '/exports/:path*',
    '/files/:path((?!.*\\.html$).*)',
  ],
};

SEO impact: Middleware that never executes for a page route cannot regress it, however the logic inside changes later.

Validation: Add a console.log in development, request a page route, and confirm nothing is logged.

Step 4: Handle hostname-wide suppression separately

Suppressing a whole preview host is a different rule with a different shape, and mixing it into the content-type logic is how production ends up noindexed.

const NON_PRODUCTION = /^(preview|staging|deploy-preview-\d+)\./;

export function middleware(request) {
  const host = request.headers.get('host') ?? '';
  if (NON_PRODUCTION.test(host)) {
    const res = NextResponse.next();
    res.headers.set('X-Robots-Tag', 'noindex, nofollow, noarchive');
    return res;
  }
  return contentTypeRules(request);
}

SEO impact: The hostname test is an allowlist of non-production patterns rather than a denylist of production ones, so an unrecognised host defaults to indexable — production behaviour, which is what you want when the rule is wrong.

Validation: curl -sI -H 'Host: preview.example.com' https://example.com/ | grep -i x-robots-tag.

Step 5: Assert the negative case in continuous integration

The dangerous failure is silent, so the test has to look for the absence of a header rather than its presence.

#!/usr/bin/env bash
# ci/assert-no-stray-noindex.sh
set -euo pipefail
fail=0
while read -r url; do
  hdr=$(curl -sI "$url" | grep -i '^x-robots-tag:' || true)
  if [ -n "$hdr" ]; then
    echo "STRAY DIRECTIVE on indexable URL: $url -> $hdr"
    fail=1
  fi
done < <(curl -s "$BASE_URL/sitemap.xml" | grep -oP '(?<=<loc>)[^<]+')
exit $fail

Validation: Temporarily broaden the matcher on a branch and confirm the job fails. A check that has never failed has not been tested.

SEO Impact Summary

The blast radius of a mistake here is proportional to how much of the site the matcher covers, which is the argument for keeping it as narrow as possible.

Matcher breadth against blast radius Three matcher scopes — specific prefixes, a broad prefix, and a site-wide matcher — with the number of pages each would suppress if the logic were wrong. named prefixes a handful of endpoints broad prefix a whole section site-wide every page, within one crawl cycle
Signal What improves What breaks if misconfigured
Index composition JSON endpoints and exports stop competing with real pages for index slots One over-broad matcher can suppress an entire section within a crawl cycle
Crawl efficiency Crawlers stop re-fetching machine-readable duplicates of page content Blocking assets needed for rendering makes pages look thin instead
Diagnostic clarity Endpoints stay crawlable and inspectable while being unindexable A Disallow instead of a header would hide them from your own auditing too
Recovery time Removing a stray header restores indexing at the next crawl Damage scales with the crawl frequency of the affected section

Measurable signals to watch:

  • Count of indexed URLs matching /api/ or /exports/ in Search Console, which should reach zero.
  • The CI assertion in Step 5, which should never fire.
  • Coverage for the sections adjacent to any matcher you change, checked for a week after the change.

Reviewing a Directive Change Before It Ships

Because the failure mode is invisible in the HTML, a middleware change deserves a specific review checklist rather than a generic one.

Pre-merge review checks for a robots header change Four sequential review gates — matcher scope, content-type guard, hostname rule, and the negative CI assertion — with the specific question each one answers about the change. matcher scope can it hit a page? type guard html short-circuits? hostname rule allowlist, not deny? CI assertion proven to fail? Every middleware change touching headers passes all four A "no" at any gate means the change ships a silent deindexing risk None of these are visible in the rendered page, so review cannot rely on a screenshot

Edge Cases and Gotchas

Content type is not always set when middleware runs On some runtimes the response headers are not populated until after the handler completes, so a content-type check inside middleware sees an empty string. Verify on your platform; if the header is unavailable, fall back to a file-extension test plus an explicit page-route exclusion.

Redirect responses do not carry useful directives A directive on a 301 is ignored because crawlers evaluate the destination. If a legacy URL must be suppressed, suppress the destination or return 410 instead — the trade-offs are covered in redirect chain management.

nofollow in a header applies to the whole response Unlike the link-level attribute, a header nofollow drops every link on the response. Fine for an API, wrong for a page you are suppressing but still want to pass equity through — use noindex, follow there.

CDN rules and middleware can both fire If your CDN also has a page rule adding headers, you have two sources for one directive and no single place to read the answer. Consolidate on one layer and delete the other, rather than documenting the interaction.

Frequently Asked Questions

Why not set X-Robots-Tag on everything and override per page? Because directives combine to the most restrictive result rather than overriding. A site-wide noindex header cannot be cancelled by a page-level index meta tag — the header wins and the page is suppressed. Default-deny is right for authentication and actively dangerous for robots directives.

Should image and font responses carry a directive? Fonts and CSS need none and must stay crawlable so Google can render the page. Images are a judgement call: a noindex removes them from image search, which is right for UI sprites and avatars and wrong for product and editorial photography that earns traffic.

Does an X-Robots-Tag on a redirect response do anything? Effectively no. Crawlers follow the redirect and apply the directives on the final response, so a directive attached to the 301 itself is ignored. Attach it to the destination if you need suppression.

How quickly does removing a stray header restore indexing? At the next crawl of each affected URL, so hours for well-linked pages and weeks for deep ones. There is no way to accelerate it beyond ensuring the URLs are in a fresh sitemap and internally linked, which is why the CI assertion matters more than the recovery plan.

Part of: Robots & Crawl Directives in Headless Stacks

Related