Robots.txt: Complete Guide and Best Practices

The robots.txt file is a plain text file at the root of your domain (e.g., yourdomain.com/robots.txt) that tells search engine crawlers which parts of your site they’re allowed — or not allowed — to access.

Basic syntax: Each rule specifies a User-agent (which crawler it applies to) and Disallow or Allow directives (which paths to block or permit). For example, blocking a staging folder from all crawlers looks like:

User-agent: *
Disallow: /staging/

Common mistakes: The most damaging error is accidentally blocking your entire site with Disallow: / after a site migration or redesign — this tells every crawler to stay away from everything. Another frequent issue is blocking CSS or JavaScript files that Google needs to render the page properly, which can hurt how your content is understood.

What robots.txt does NOT do: It doesn’t remove pages from Google’s index — a blocked page can still appear in search results (usually without a description) if other sites link to it. To fully keep a page out of search results, use a noindex meta tag instead, on a page that robots.txt allows crawlers to reach.

Best practices: Keep the file simple, always include your sitemap URL at the bottom, test changes in Search Console’s robots.txt tester before deploying, and audit it after every major site migration.