Back to Glossary
What Is Crawl/Crawling?
SEO
Updated 18 September 2026
Quick Answer
Crawling is the process by which search engines automatically discover and scan webpages by following links, building the index they use to serve search results.
Crawling vs. Indexing — What's the Difference?
| Crawling | Indexing | |
|---|---|---|
| What happens | A search engine discovers and reads a page | The page is stored and made eligible to appear in search results |
Why It Matters
- If a page can't be crawled, it can't be indexed or ranked, regardless of how good its content is.
- Technical issues (broken links, blocked pages) can silently prevent crawling without any obvious visible symptom.
How It Works
- A search engine's automated crawler visits a page.
- It follows the links on that page to discover more pages.
- Discovered pages are evaluated for potential indexing.
Key Takeaways
- Crawling is how search engines discover pages by following links.
- A page that can't be crawled can't be indexed or ranked at all.
- Google Search Console shows crawl activity and errors directly.
Frequently Asked Questions
How do I know if my site is being crawled properly?
Google Search Console shows crawl activity and any errors preventing it directly.
Can I stop certain pages from being crawled?
Yes, using a robots.txt file or meta tags — useful for pages you don't want appearing in search results.
Why would a page not get crawled?
Common causes include broken internal links, a missing or broken sitemap, or accidental blocking via robots.txt.