Back to Glossary
What Is Robots.txt?
Updated 18 September 2026
Quick Answer
A robots.txt file is a text file on a website that tells search engine crawlers which pages or sections they're allowed or not allowed to crawl.
Robots.txt vs. Nofollow — What's the Difference?
| Robots.txt | Nofollow | |
|---|---|---|
| Scope | Applies to entire pages or sections of a site | Applies to individual links |
Why It Matters
- An incorrectly configured robots.txt file can accidentally block search engines from crawling important pages entirely, a serious, often-overlooked technical SEO issue.
- It's genuinely worth checking if a page mysteriously isn't appearing in search results at all.
How It Works
- A text file is placed at the root of a website (yoursite.com/robots.txt).
- It specifies rules for which crawlers can access which parts of the site.
- Search engines read this file before crawling, respecting its instructions in most cases.
Key Takeaways
- A robots.txt file tells search engine crawlers which pages they can or can't access.
- A misconfigured one can accidentally block important pages from being crawled entirely.
- It's worth checking directly if pages mysteriously aren't appearing in search.
Frequently Asked Questions
Do I need to create a robots.txt file myself?
Most website platforms generate a basic one automatically — it's worth checking it exists and isn't accidentally blocking important content.
How do I check my robots.txt file?
Visit yourdomain.com/robots.txt directly in a browser to see its current rules.
Can a robots.txt mistake hurt my SEO?
Yes significantly — accidentally blocking your entire site or key sections is a real, sometimes overlooked cause of pages not appearing in search at all.