A root-level file that tells crawlers which URLs they may or may not fetch.
Robots.txt is a plain-text file at the root of a domain that gives crawling instructions to bots via allow and disallow rules. It controls what search engines are permitted to crawl, and it is the first file most crawlers request. It is a crawling directive, not an indexing one.
A critical distinction: disallowing a URL stops Google fetching its content, but the URL can still be indexed (without a snippet) if it is linked from elsewhere. To keep a page out of the index, use a noindex tag, which requires the page to be crawlable.
A single wrong line can be catastrophic. Disallow: / blocks the entire site from crawling, which is a classic launch-day disaster when a staging rule ships to production. Robots.txt is powerful precisely because it is applied site-wide.
Using robots.txt to try to hide a page from search (use noindex instead), blocking CSS or JS that Google needs to render the page, or leaving a staging Disallow: / rule live after go-live.
Common questions
Straight answers on how this fits your marketing and build.
No. Blocking a URL stops it being crawled, but a blocked URL can still be indexed without a snippet if it is linked. To deindex, allow crawling and add a noindex tag, or remove the page.
Still have questions? Talk to a specialist
Get a free audit and a 6-month roadmap showing exactly where you stand vs your top 3 competitors — no fluff, no upsell.
Trusted by 500+ brands · 4.9/5 client rating