What is Robots.txt

Common mistakes

Using robots.txt to try to hide a page from search (use noindex instead), blocking CSS or JS that Google needs to render the page, or leaving a staging Disallow: / rule live after go-live.

• Do not block resources needed for rendering
• Do not rely on it to deindex pages
• Reference your sitemap with a Sitemap: line

Why it matters

robots.txt is the first file Googlebot requests on any site, and a single wrong line can be catastrophic — a stray `Disallow: /` pushed to production has taken major sites out of Google entirely. It is a crawl-control tool, not an index-control tool, and confusing the two is the most common and damaging mistake. Used well, it conserves crawl budget and keeps bots out of admin, search-result and infinite URL spaces; used carelessly, it silently blocks the pages you most want ranked.

• The first file every crawler checks — errors here affect the whole site
• Controls crawling, not indexing — a blocked URL can still appear in results
• A wrong Disallow can de-rank an entire site, so it needs careful review

Common mistakes and the noindex trap

The classic error is using robots.txt to try to remove a page from Google. Because a disallowed page is never crawled, Google never sees a noindex tag on it — so a page you block in robots.txt AND noindex can stay indexed indefinitely, since the crawler can never read the noindex. To reliably de-index a page you must allow it to be crawled and serve a noindex (or use the removals tool). Also remember robots.txt is public: never rely on it to hide sensitive URLs, since you are literally publishing their paths.

• To de-index a page, do NOT block it in robots.txt — it must be crawlable to read the noindex
• Blocking CSS/JS can break rendering and hurt how Google sees the page
• robots.txt is publicly readable — never use it as a security measure

In practice

A SaaS company launches a redesign and organic traffic collapses overnight. The cause is a single line — `Disallow: /` — carried over from the staging robots.txt and pushed live, blocking Googlebot from the entire site. Because the pages were still indexed but now uncrawlable, they began dropping out of results as Google refreshed. The immediate fix is to correct robots.txt to allow crawling, then use URL Inspection to request re-crawling of key pages. The broader lesson: robots.txt is the highest-blast-radius file on a site, so it belongs in code review and deploy checks, staging and production versions must never be confused, and after any deploy the live robots.txt should be verified in Search Console's robots.txt report before you move on.