What is noindex?
Noindex is an instruction that keeps a page out of search results. Google says that when it finds the rule, it "will drop that page entirely from Google Search results, regardless of whether other sites link to it."
It goes in one of two places:
- A robots meta tag in the page's HTML:
<meta name="robots" content="noindex"> - An HTTP response header:
X-Robots-Tag: noindex, useful for files such as PDFs.
Why doesn't noindex work with robots.txt?
Because Google has to read the page to see the rule. Google warns: "If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results." See what is robots.txt.
Google also says noindex inside robots.txt "is not supported."
When should a moving company use noindex?
For pages that shouldn't appear in search, such as thank-you pages, internal tools, or test and staging sites. The most common mistake runs the other way: a staging site's noindex is copied to the live site at launch. Google's site-move guidance says not to forget "to remove any noindex or robots.txt blocks that were only needed for the migration." See how to redesign a website without losing rankings.