In one sentence: Disallow in robots.txt stops a crawler fetching a page, noindex tells it not to keep the page — and because a disallowed page is never fetched, its noindex is never read, which is the trap.
The trap, precisely
You want a section out of the index. You add Disallow: /internal/ to robots.txt. The crawler stops fetching those pages — and therefore never sees the noindex you also added. Pages already in the index stay there, sometimes indefinitely, sometimes listed with no description because the crawler is no longer allowed to look.
To remove a page from the index, allow the crawl and servenoindex. To keep a crawler out of a section it was never going to index anyway, useDisallow. Doing both to the same URL achieves the opposite of what you wanted.
Which mechanism for which case
| Goal | Mechanism |
|---|---|
| Remove an HTML page from the index | <meta name="robots" content="noindex">, crawl allowed |
| Remove a PDF, image or API response | X-Robots-Tag: noindex header — there is no head to put a meta tag in |
| Keep crawlers out of an admin path entirely | Disallow, accepting it will not de-index anything already there |
| Keep a whole subdomain out | Its own robots.txt with Disallow: /, plus noindex at the app level |
The subdomain nobody checks
Every robots.txt is per-host. Your marketing site's file says nothing about your app, your docs, your staging environment or your CDN host. Each one needs its own, and the common failure is that the app subdomain has no robots.txt at all — a request for it returns the application's own 404 page, which is not a robots file, so the subdomain is fully open.
We found exactly this on our own product subdomain: no robots.txt, and the login page already picking up impressions. A product back-end has no reason to be in search results, and it dilutes the signals of the domain you are actually working on.
Belt and braces, deliberately
For a surface that should never be indexed, use both: a robots.txt with Disallow: / to stop the crawl, and application-level noindex to cover the case where a page gets linked from elsewhere and fetched anyway. The two cover different failure modes, and the combination is the one place where doing both is correct rather than contradictory — because here you do not need de-indexing, you need to never be indexed in the first place.
Verify by response, not by config
Request the URL and read the actual headers and the actual HTML. A framework's metadata API, a CDN rule and an origin config can each override the others, and the only thing that settles it is the response the crawler receives.