Controlling Facet URL Explosion in Headless Catalogs

A filter component that renders each option as an anchor turns a catalogue of forty thousand products into a crawlable space of several million URLs, none of which anybody decided to publish.

When to Use This Approach

Apply this when:

  • Verified crawler logs show that most Googlebot requests carry a query string.
  • Search Console reports far more discovered URLs than your sitemap contains.
  • New product pages take weeks to be indexed while filter combinations are crawled repeatedly.

Measuring Before Cutting

The surface has two sizes: the theoretical one your UI can produce, and the discovered one Googlebot has actually found. The gap between them tells you how much worse the problem is still going to get.

Theoretical surface, discovered URLs, and the indexable set Three nested magnitudes drawn as bars — the theoretical combinatorial surface, the subset Googlebot has discovered so far, and the small allowlist that is actually intended to be indexed. One catalogue, three very different numbers theoretical surface 3,100,000 URLs the UI can emit discovered so far 418,000 URLs Googlebot has requested intended to be indexed 340 allowlisted combinations still climbing Discovery grows until the crawl paths are removed — canonicals alone do not slow it

Implementation Steps

Step 1: Quantify the reachable surface

// scripts/facet-surface.mjs
import { facets } from '../lib/catalog.mjs';

const categories = 42;
const values = facets.map((f) => f.values.length);

let reachable = categories;                       // bare category pages
for (let depth = 1; depth <= facets.length; depth++) {
  // combinations of `depth` facets, each contributing its value count
  reachable += categories * combinations(values, depth);
}
console.log(`${reachable.toLocaleString()} reachable URLs`);

function combinations(counts, k, start = 0) {
  if (k === 0) return 1;
  let total = 0;
  for (let i = start; i <= counts.length - k; i++) {
    total += counts[i] * combinations(counts, k - 1, i + 1);
  }
  return total;
}

Validation: Compare against the count of distinct parameterised URLs in a month of verified crawler logs. If the discovered number is still rising month over month, discovery is ongoing and the paths are still open.

Step 2: Find the crawl paths

# Every crawlable facet link the listing page emits
curl -s https://example.com/shop/shoes/ \
  | grep -oE 'href="[^"]*\?[^"]*"' \
  | sed 's/href="//; s/"$//' \
  | awk -F'?' '{print $2}' \
  | tr '&' '\n' | cut -d= -f1 | sort | uniq -c | sort -rn

Validation: Every parameter name in that output is a dimension the crawler can multiply. A parameter that appears in links but is not on your allowlist is a crawl path to close.

Step 3: Convert non-indexable controls to buttons

The visitor’s experience is unchanged; only the markup differs.

// components/FacetOption.jsx
import { useRouter } from 'next/navigation';
import { isIndexableCombination } from '@/lib/indexable-facets';

export function FacetOption({ facet, value, active, currentParams }) {
  const router = useRouter();
  const next = toggle(currentParams, facet, value);

  if (isIndexableCombination(next)) {
    return (
      <a href={pathFor(next)} aria-pressed={active}>
        {value}
      </a>
    );
  }

  return (
    <button
      type="button"
      aria-pressed={active}
      onClick={() => router.replace(queryFor(next), { scroll: false })}
    >
      {value}
    </button>
  );
}

SEO impact: A crawler parsing the page finds only the allowlisted facet URLs, so the combinatorial space is never discovered. Visitors keep every filter.

Validation: curl -s https://example.com/shop/shoes/ | grep -c 'href="[^"]*?' should drop to the allowlist size plus pagination.

Step 4: Keep the URL shareable without making it crawlable

Visitors legitimately want to send a filtered view to a colleague. Updating the address bar through the history API preserves that without emitting an anchor.

// lib/facet-url.js
export function applyFacetState(state) {
  const params = new URLSearchParams();
  for (const [facet, values] of Object.entries(state)) {
    for (const v of values) params.append(facet, v);
  }
  const qs = params.toString();
  window.history.replaceState(null, '', qs ? `?${qs}` : location.pathname);
}

SEO impact: The URL still works when pasted — the server renders the filtered view — but nothing in your own HTML links to it, so it enters the crawl frontier only if someone publishes it externally.

Validation: Copy the URL after applying filters, open it in a private window, and confirm the filtered results render server-side.

Step 5: Handle the URLs already discovered

Removing links stops new discovery; it does not un-discover what Google already has.

// The canonical for a non-allowlisted combination points at its nearest ancestor
export async function generateMetadata({ params, searchParams }) {
  const combination = contentParams(searchParams);
  const indexable = isIndexableCombination(combination);
  return {
    alternates: { canonical: nearestIndexableAncestor(params, combination) },
    robots: indexable ? { index: true, follow: true } : { index: false, follow: true },
  };
}

SEO impact: The already-discovered URLs stop competing immediately, and their crawl frequency decays over subsequent cycles as Google learns they are neither canonical nor indexable.

Validation: Re-measure parameterised crawl share monthly using the method in parsing CDN logs for Googlebot crawl waste.

What Changes for Each Party

The change is asymmetric by design: crawlers lose a great deal, visitors lose almost nothing.

Effect on visitors compared with effect on crawlers Two rows — visitor capabilities and crawler discovery — each showing what is retained and what is removed after non-allowlisted facet links become buttons. party retained removed visitor every filter, back button, shareable URL, server render open-in-new-tab on filters crawler every allowlisted facet page, every product, all pagination millions of combination URLs that never earned a visit The asymmetry is the whole argument for the change

SEO Impact Summary

Reclaimed crawl capacity does not vanish — it moves to the URLs that were previously waiting behind the combinations.

Where the reclaimed crawl requests go Two stacked bars showing crawl allocation before and after removing facet crawl paths, with the parameter share transferring to products and allowlisted facets. before products parameter combinations after products facets Total requests barely change — what they are spent on changes completely
Signal What improves What breaks if misconfigured
Crawl budget Requests shift from combination URLs to products and allowlisted facets Converting an allowlisted facet to a button removes a page you wanted indexed
Discovery speed New products are found in days rather than weeks Removing pagination anchors alongside filters strands deep inventory
Index composition The indexed set converges on the sitemap set Leaving canonicals off the remaining discovered URLs keeps duplicates competing
Rendering cost Fewer uncached parameter combinations reaching the origin Client-only filtering removes the server-rendered result set entirely

Measurable signals to watch:

  • Share of verified crawler requests carrying a query string, tracked monthly.
  • Count of distinct parameterised URLs seen in logs, which should plateau then decline.
  • Time from product publication to first crawl, which is the outcome the freed budget buys.

Edge Cases and Gotchas

Pagination is not a facet It is tempting to convert pagination to buttons in the same change. Do not — pagination anchors are the crawl path to deep inventory, and removing them is how catalogues lose their tail. The distinction is worked through in crawlable pagination without rel next prev.

Buttons must still be accessible A <button> with aria-pressed communicates a toggle correctly; a <div> with a click handler does not. Keyboard operation and screen-reader semantics are not optional side effects of this change.

Server-side rendering of the filtered view must be preserved If removing the anchor also moves filtering to the client, a shared URL renders an empty shell for anyone without JavaScript — including the crawler that eventually finds it externally. Keep the server render; only the link is removed.

Some facets deserve promotion, not suppression A facet with genuine search demand should become a real path-segment URL rather than a suppressed query string. Deciding which is the subject of deciding which facet combinations to index.

Frequently Asked Questions

Does converting filters to buttons hurt usability? Not when implemented properly. A button that updates state and pushes a history entry gives the visitor a working back button and a shareable URL while emitting no crawlable anchor. The only real loss is open-in-new-tab, which still works on the allowlisted facets you keep as links.

Is nofollow enough to stop facet crawling? No. Google treats it as a hint for crawl scheduling, and the URL remains discoverable from the markup, from other sites, and from earlier crawls. Removing the anchor is the only reliable way to stop discovery.

How long before crawl volume actually drops? One to several crawl cycles. Google re-requests URLs it already knows for some time after the links disappear, so expect a gradual decline over weeks and measure the trend rather than the day after.

What if the filter UI is a third-party search component? Most hosted search components expose a link-rendering option or a template hook. If yours does not, wrap its output and rewrite anchors to buttons in the adapter layer — the constraint is worth solving rather than accepting, because a component you cannot control is emitting your crawl paths.

Part of: Faceted Navigation & URL Parameters

Related