Indexing and canonicals

The four layers are the robots file, the page metadata, the sitemap, and the response headers. Each can suppress indexing on its own, and each is configured in a different place by a different concern. Verification checks all four and asserts they agree.

The response header is the trap. A header outranks every page's metadata, applies to responses the page metadata cannot reach — including image bytes — and is invisible to anyone inspecting the rendered document. A site whose pages all declare themselves indexable can still be entirely suppressed by a single setting applied during testing and never removed.

Sites here default to indexable, and the setting is an opt-out used while testing, not an opt-in that production depends on somebody remembering. A site that has to remember to switch indexing on is a site that will eventually ship without it, and the failure is silent for as long as nobody checks.

Every page declares its own canonical. No canonical is set on the root layout, because the not-found page inherits root metadata and would then declare every missing address on the site to be a duplicate of the homepage.

The sitemap is generated from the same route data the pages are generated from, so it cannot list a page that does not exist or omit one that does.

The robots file publishes the sitemap location instead of leaving it to be discovered. A sitemap that exists but is announced nowhere is a file on a server, and the cost of the line that announces it is one line.

Test environments are suppressed by a mechanism that cannot survive being promoted to live. Suppression driven by a setting disappears when the setting does; suppression written into a committed file travels with the code and is still there months later, quietly keeping the live site out of every index.

After release, indexing is verified against the live site instead of against the build status, because those are different claims and only one of them is what a search engine sees.