If Googlebot can’t reliably crawl your site, none of your other SEO work matters — content quality, backlinks, and keyword targeting are all irrelevant for pages Google can’t successfully access and process. Crawl errors are one of the least glamorous parts of technical SEO, but they’re also among the highest-leverage issues to fix, because a single crawl or indexing problem can silently remove entire sections of a site from search results.
Where to Find Crawl Errors in Search Console
Google Search Console consolidates crawl and indexing information primarily in two reports: the Pages report (under Indexing), which shows why individual URLs are or aren’t indexed, and the Crawl Stats report (under Settings), which shows request volume, response codes, and file types Googlebot is requesting over time. Between the two, you can usually diagnose most crawling problems without needing third-party tools.
The Most Common Crawl Errors and What They Mean
404 (Not Found)
Googlebot requested a URL that no longer exists. Some 404s are expected and harmless — old products, expired promotions — but a spike in 404s, especially on URLs that used to rank, usually signals a broken migration, a faulty redirect map, or internal links pointing to deleted pages.
Fix: For pages with real replacement content, add a 301 redirect to the closest equivalent URL. For pages with no reasonable equivalent, a clean 404 (or 410 for permanently removed content) is fine — just make sure no internal links still point to the dead URL.
Server Errors (5xx)
These indicate your server failed to respond correctly when Googlebot requested a page — often due to hosting resource limits, misconfigured caching, or the server being overwhelmed during a crawl spike.
Fix: Check server logs around the timestamps Search Console reports, review hosting resource limits, and consider whether a spike in bot traffic (not just Googlebot) needs to be throttled via rate limiting rather than blocking Google outright.
“Discovered – currently not indexed” and “Crawled – currently not indexed”
These two statuses are increasingly common and often misunderstood. “Discovered” means Google knows the URL exists but hasn’t crawled it yet, usually due to crawl budget prioritization. “Crawled – currently not indexed” means Google visited the page but chose not to add it to the index — frequently a quality or duplication signal rather than a technical error.
Fix: For discovered-not-crawled pages, improve internal linking so important pages are easier for Googlebot to find and prioritize. For crawled-not-indexed pages, the more common cause is thin, duplicate, or low-value content rather than a crawling problem — review whether the page offers something genuinely distinct before assuming it’s a technical issue.
Redirect Errors
Redirect chains (A → B → C) and redirect loops (A → B → A) waste crawl budget and can cause Google to give up before reaching the final destination.
Fix: Audit redirects periodically and flatten chains so every redirect points directly to its final destination in a single hop.
Blocked by robots.txt
A URL Google wants to crawl is disallowed in your robots.txt file. This is sometimes intentional (staging environments, internal search results) but often accidental, especially after a site migration where a “disallow all” rule from a staging environment gets carried into production.
Fix: Review robots.txt directives against the list of blocked URLs, and remove any disallow rules blocking pages you actually want indexed.
Blocked Due to Other 4xx Issue / Soft 404
A soft 404 is a page that returns a 200 (success) status code but whose content looks like an error page to Google — for example, an empty product page after a product is discontinued.
Fix: Either restore meaningful content to the page, return a proper 404/410 status code, or redirect to a relevant replacement page.
A Step-by-Step Workflow for Cleaning Up Crawl Errors
- Open the Pages report and review the “Why pages aren’t indexed” breakdown for patterns rather than fixing URLs one at a time.
- Cross-reference spikes in the Crawl Stats report with recent deploys, migrations, or hosting changes.
- Group errors by root cause (a broken template, a bad redirect rule, a robots.txt mistake) rather than treating each URL as an isolated issue.
- Fix the root cause, then use the URL Inspection tool to test a sample of affected URLs.
- Request indexing for a small batch of high-priority fixed URLs rather than the entire affected set at once.
Frequently Asked Questions
Do crawl errors directly hurt my rankings?
Not automatically — a handful of 404s from naturally expired content is normal for any active site. The risk is when errors affect pages that should be indexed and ranking, effectively removing them from search results entirely.
How often should I check the Pages and Crawl Stats reports?
Monthly for most sites, and immediately after any migration, redesign, or major content restructuring, since those are the events most likely to introduce widespread crawl issues.
Should I use the “Request Indexing” tool for every fixed page?
It’s most useful for a small number of high-priority pages. For large-scale fixes, submitting an updated XML sitemap and letting Google recrawl naturally is more efficient than manually requesting indexing for hundreds of URLs.
Final Thoughts
Crawl errors are rarely exciting, but they’re some of the most reliable wins in technical SEO because fixing them removes obstacles rather than requiring you to create anything new. A regular audit rhythm — monthly checks plus post-migration reviews — catches most problems before they meaningfully affect visibility.
References
- Google Search Central — Search Console Getting Started Guide
- Google Search Central — Introduction to robots.txt
- Google Search Central — HTTP and network errors documentation
Need help auditing and cleaning up your site’s crawl and indexing issues? [Link to your contact/services page here.]