Robots.txt

The robots.txt file is a simple text file placed at the root of your website that tells search engine crawlers which pages or directories they’re allowed to access. It’s part of the Robots Exclusion Protocol and is crucial for managing crawl budgets, protecting sensitive content, and avoiding indexing duplicate or irrelevant pages.

Learn how to set it up correctly in our guide: How to Use robots.txt: The Original Way to Tell Crawlers What to Do