Robots.txt: Complete Guide and Best Practices
The robots.txt file is a plain text file at the root of your domain (e.g., yourdomain.com/robots.txt) that tells search engine crawlers which parts of your site they’re allowed — or not allowed — to access.
Basic syntax: Each rule specifies a User-agent (which crawler it applies to) and Disallow or Allow directives (which paths to block or permit). For example, blocking a staging folder from all crawlers looks like:
User-agent: *
Disallow: /staging/
Common mistakes: The most damaging error is accidentally blocking your entire site with Disallow: / after a site migration or redesign — this tells every crawler to stay away from everything. Another frequent issue is blocking CSS or JavaScript files that Google needs to render the page properly, which can hurt how your content is understood.
What robots.txt does NOT do: It doesn’t remove pages from Google’s index — a blocked page can still appear in search results (usually without a description) if other sites link to it. To fully keep a page out of search results, use a noindex meta tag instead, on a page that robots.txt allows crawlers to reach.
Best practices: Keep the file simple, always include your sitemap URL at the bottom, test changes in Search Console’s robots.txt tester before deploying, and audit it after every major site migration.
Monika Gupta
Digital Marketring Executive
ClicZeo Editorial Team is a team of digital marketing professionals specializing in SEO, AEO (Answer Engine Optimization), GEO, Google Ads, Meta Ads, content marketing, and local business growth. We create data-driven content to help businesses improve their online visibility, generate qualified leads, and stay ahead of the latest digital marketing trends.
Found this useful?
Add ClicZeo as a Preferred Source on Google and our guides on AI search, SEO and performance marketing will show up first when you search our topics.
Set ClicZeo as a Preferred SourceOpens Google’s settings — search for “ClicZeo” and add it. Takes a few seconds.