What is robots.txt?

Robots.txt is a text file at the root of a website that tells crawlers which URLs they may visit, and Google says it manages crawling but doesn't keep a page out of search results.

GlossaryReviewed by Nicholas DiMoro

Learning objectives

After reading this article you will be able to:

Define robots.txt and what it controls
Explain what robots.txt can't do

What is robots.txt?

Robots.txt is a plain text file at the root of a website, such as https://www.example.com/robots.txt. Google says it "tells search engine crawlers which URLs the crawler can access on your site."

A simple file looks like this:

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Each group names a crawler (User-agent) and the paths it may not visit. The Sitemap line points crawlers to the site's XML sitemap.

What can't robots.txt do?

Keep a page out of Google. Google says robots.txt "is not a mechanism for keeping a web page out of Google." Use noindex or password protection instead.
Force crawlers to comply. Google says "it's up to the crawler to obey them," and while reputable crawlers do, "other crawlers might not."

Why does robots.txt matter for a moving company?

One wrong line can block search engines from the whole site, for example a rule left over from a staging site. Robots.txt is also where a business allows or blocks specific AI crawlers, such as the ones ChatGPT and Perplexity use for search. See what are AI crawlers.

FAQs

What does blocked by robots.txt mean?: It means the site's robots.txt file tells Google's crawler not to visit that URL. Google won't crawl it, but the URL can still appear in search results if other pages link to it.
Can robots.txt be ignored?: Yes. Google says robots.txt can't enforce crawler behavior. Googlebot and other reputable crawlers follow it, but other crawlers might not.

Sources