A root-level file that tells crawlers which URLs they may or may not fetch.
Robots.txt is a plain-text file at the root of a domain that gives crawling instructions to bots via allow and disallow rules. It controls what search engines are permitted to crawl, and it is the first file most crawlers request. It is a crawling directive, not an indexing one.
A critical distinction: disallowing a URL stops Google fetching its content, but the URL can still be indexed (without a snippet) if it is linked from elsewhere. To keep a page out of the index, use a noindex tag, which requires the page to be crawlable.
A single wrong line can be catastrophic. Disallow: / blocks the entire site from crawling, which is a classic launch-day disaster when a staging rule ships to production. Robots.txt is powerful precisely because it is applied site-wide.
Using robots.txt to try to hide a page from search (use noindex instead), blocking CSS or JS that Google needs to render the page, or leaving a staging Disallow: / rule live after go-live.
robots.txt is the first file Googlebot requests on any site, and a single wrong line can be catastrophic — a stray `Disallow: /` pushed to production has taken major sites out of Google entirely. It is a crawl-control tool, not an index-control tool, and confusing the two is the most common and damaging mistake. Used well, it conserves crawl budget and keeps bots out of admin, search-result and infinite URL spaces; used carelessly, it silently blocks the pages you most want ranked.
The classic error is using robots.txt to try to remove a page from Google. Because a disallowed page is never crawled, Google never sees a noindex tag on it — so a page you block in robots.txt AND noindex can stay indexed indefinitely, since the crawler can never read the noindex. To reliably de-index a page you must allow it to be crawled and serve a noindex (or use the removals tool). Also remember robots.txt is public: never rely on it to hide sensitive URLs, since you are literally publishing their paths.
A SaaS company launches a redesign and organic traffic collapses overnight. The cause is a single line — `Disallow: /` — carried over from the staging robots.txt and pushed live, blocking Googlebot from the entire site. Because the pages were still indexed but now uncrawlable, they began dropping out of results as Google refreshed. The immediate fix is to correct robots.txt to allow crawling, then use URL Inspection to request re-crawling of key pages. The broader lesson: robots.txt is the highest-blast-radius file on a site, so it belongs in code review and deploy checks, staging and production versions must never be confused, and after any deploy the live robots.txt should be verified in Search Console's robots.txt report before you move on.
Part of our defined terms knowledge graph — browse every entry in this branch.
How many pages a search engine will crawl on your site, and how often.
Whether a page is stored in a search engine’s index and eligible to rank.
A machine-readable list of your important URLs, submitted to help discovery.
The underlying goal behind a search query, which content must satisfy to rank.
The Search Engine Results Page — everything a search engine returns for a query.
Common questions
Straight answers on how this fits your marketing and build.
Still have questions? Talk to a specialist