Generating robots.txt From CMS Content

A hand-maintained robots.txt describes the routes a site had when someone last edited it, which is rarely the routes it has today.

When to Use This Approach

Generate the file rather than maintaining it when:

• Routes are created from CMS content, so the set of paths changes without a code change.
• More than one team can add routes, and nobody owns the file that describes them.
• You have found at least one rule that blocks a path which no longer exists — a reliable sign that the reverse error is also present somewhere.

Three Inputs, One Output

The file has exactly three legitimate sources. Anything else in it is a rule somebody added during an incident and never removed.

Implementation Steps

Step 1: Audit what the current file blocks

Before generating anything, find out how far the existing file has drifted.

# Which Disallow rules no longer match any built route?
curl -s https://example.com/robots.txt \
  | grep -oP '(?<=^Disallow: ).*' > rules.txt

find _site -name index.html | sed 's|_site||; s|/index.html|/|' | sort > routes.txt

while read -r rule; do
  pat=$(printf '%s' "$rule" | sed 's/\*/.*/g')
  grep -qE "^$pat" routes.txt || echo "stale rule: $rule"
done < rules.txt

Validation: Every stale rule is dead weight. More importantly, the same script run in reverse tells you which live routes a rule catches — run it before deleting anything.

Step 2: Model the inputs

// lib/robots-inputs.mjs
export const INVARIANT = [
  { userAgent: '*', disallow: ['/api/', '/_next/data/'] },
];

export async function editorExclusions(cms) {
  const settings = await cms.settings.get('seo');
  // A validated list of path prefixes, not a free-text blob.
  return (settings.crawlExclusions ?? [])
    .map((p) => p.trim())
    .filter((p) => p.startsWith('/') && !p.includes('\n'));
}

export function routeExclusions(manifest) {
  return manifest.routes
    .filter((r) => r.meta?.crawl === 'deny')
    .map((r) => r.pattern.replace(/\[\.\.\.[^\]]+\]/g, '*').replace(/\[[^\]]+\]/g, '*'));
}

SEO impact: Editor control is preserved without giving anyone a text field that writes directly to a production crawl directive.

Validation: Feed the CMS a deliberately malformed exclusion and confirm the filter drops it rather than emitting it.

Step 3: Generate the file

// scripts/build-robots.mjs
import { writeFileSync } from 'node:fs';
import manifest from '../.build/route-manifest.json' with { type: 'json' };
import { cms } from '../lib/cms.mjs';
import { INVARIANT, editorExclusions, routeExclusions } from '../lib/robots-inputs.mjs';

const host = process.env.SITE_HOST ?? 'https://example.com';
const isProduction = process.env.DEPLOY_ENV === 'production';

const lines = [];
if (!isProduction) {
  lines.push('User-agent: *', 'Disallow: /');
} else {
  const disallow = [
    ...INVARIANT.flatMap((r) => r.disallow),
    ...routeExclusions(manifest),
    ...(await editorExclusions(cms)),
  ];
  lines.push('User-agent: *');
  for (const path of [...new Set(disallow)].sort()) lines.push(`Disallow: ${path}`);
  lines.push('', `Sitemap: ${host}/sitemap.xml`);
}

writeFileSync('public/robots.txt', lines.join('\n') + '\n');
console.log(lines.join('\n'));

SEO impact: Non-production deployments block everything by default, so a preview host can never be crawled because someone forgot a setting.

Validation: Run the script with DEPLOY_ENV unset and confirm the output is a blanket disallow.

Step 4: Validate against the indexable URL set

The rule that matters is not whether the syntax is valid but whether it blocks something you want indexed.

// scripts/validate-robots.mjs — simulate the rules over the sitemap
import { readFileSync } from 'node:fs';

const rules = readFileSync('public/robots.txt', 'utf8')
  .split('\n')
  .filter((l) => l.startsWith('Disallow: '))
  .map((l) => l.slice(10).trim())
  .filter(Boolean)
  .map((p) => new RegExp('^' + p.replace(/[.+?^{}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*')));

const urls = readFileSync('.build/sitemap-urls.txt', 'utf8').trim().split('\n');
const blocked = urls.filter((u) => rules.some((r) => r.test(new URL(u).pathname)));

if (blocked.length) {
  console.error(`robots.txt blocks ${blocked.length} sitemap URL(s):`);
  blocked.slice(0, 20).forEach((u) => console.error('  ' + u));
  process.exit(1);
}
console.log(`robots.txt validated against ${urls.length} indexable URLs`);

SEO impact: Catches the only genuinely expensive robots.txt mistake — a rule broader than intended — before it reaches a crawler.

Validation: Add Disallow: /blog on a branch and confirm the build fails with a list of blocked article URLs.

Step 5: Serve it with a short cache lifetime

// next.config.mjs
export default {
  async headers() {
    return [{
      source: '/robots.txt',
      headers: [{ key: 'Cache-Control', value: 'public, max-age=300, s-maxage=300' }],
    }];
  },
};

Validation: curl -sI https://example.com/robots.txt | grep -i cache-control. Five minutes is long enough to absorb crawler re-fetches and short enough that an emergency fix propagates before the next crawl cycle.

SEO Impact Summary

Generating the file also closes the loop with the sitemap, since both are derived from the same route data and can be validated against each other.

Measurable signals to watch:

• Search Console’s robots.txt report status, checked after every deploy that changes routes.
• The count of stale rules from Step 1, which should stay at zero once generation is in place.
• Crawl requests to excluded prefixes, which should fall to near zero within a crawl cycle.

Failure Modes Ranked by Cost

Not every mistake in this file costs the same. Knowing the ordering tells you where to put the validation effort.

The bottom row is why Step 4 exists. The first three failures are cosmetic or self-healing; the fourth removes a section from search until somebody notices the traffic drop.

Edge Cases and Gotchas

Blocking asset directories breaks rendering, not just assets A Disallow: /_next/ looks harmless but stops Google fetching the JavaScript and CSS needed to render the page, so the rendered view collapses to an unstyled shell. Never block the framework’s asset directory, however much crawl traffic it attracts.

Wildcards match more than the path segment you were thinking of Disallow: /search* also matches /search-results-guide/, an article you may well want indexed. Anchor rules with a trailing slash where you mean a directory.

The file is per origin, including per subdomain and per scheme https://example.com/robots.txt says nothing about https://shop.example.com/. Generate one per host, and remember that a hostname introduced by a new market or a new brand starts with no file at all.

Crawl-delay is ignored by Google It is honoured by some other crawlers, so it is not useless, but it will not slow Googlebot. If crawl volume is the problem, the answer is fewer crawlable URLs — see faceted navigation & URL parameters — or the crawl rate setting, not this file.

Frequently Asked Questions

Should editors be able to change robots.txt? They should control a narrow, validated slice — typically a list of path prefixes to exclude — through a settings entry rather than a free-text field. A free-text robots.txt in the CMS is a production configuration file with no review, no test, and no rollback.

How do I handle robots.txt for multiple hostnames? Each hostname needs its own file, since crawlers fetch it per origin. Generate per host from the same inputs, with non-production hosts emitting a blanket disallow. One shared file across hosts means either production is over-blocked or preview is exposed.

Does a syntax error block the whole site? Unparseable lines are ignored, so typos usually fail open. The dangerous case is a well-formed rule broader than intended — a trailing wildcard on a prefix that also matches real pages parses perfectly and blocks exactly what it says.

Can I use robots.txt to remove a page from search? No, and this is the most common misconception about the file. Blocking the crawl also blocks the noindex a crawler would need to read, so the URL can persist in results as a bare link. The distinction is worked through in noindex vs robots Disallow for headless routes.

Part of: Robots & Crawl Directives in Headless Stacks

Related

• Serving X-Robots-Tag From Edge Middleware — the per-response half of the same control surface
• XML Sitemap Generation for Headless — the file this one points at, and the URL set it is validated against
• Automating Dynamic Route Generation for Headless Blogs — where the route manifest that drives this generator comes from