The Complete robots.txt Guide: Common Mistakes and Advanced Directives (2026)

The robots.txt file is one of the oldest standards on the web, and yet it remains one of the most misunderstood. In technical SEO audits, a broken or overly aggressive robots.txt is still one of the most common reasons a site underperforms — sometimes for months before anyone notices. In 2026, the file has taken on a second life: it is now the front door not only for Googlebot and Bingbot, but for an entire generation of AI crawlers deciding whether your content appears in AI-generated answers.

This guide covers how robots.txt actually works, the mistakes we see most often in audits, and the advanced directives worth knowing. If you want to generate a clean starter file, you can use our free robots.txt & sitemap.xml generator and then customize it using the rules below.

What robots.txt Actually Does (and What It Doesn’t)

The robots.txt file lives at the root of your domain — always at https://yourdomain.com/robots.txt, never in a subdirectory — and gives crawl instructions to compliant bots. Before a well-behaved crawler fetches any URL on your site, it checks this file to see which paths it is allowed to request.

Here is the crucial distinction that trips up even experienced marketers: robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in Google’s index if other sites link to it. Google simply indexes the URL without content, producing the infamous “No information is available for this page” result. If your goal is to keep a page out of search results entirely, the correct tool is a noindex meta tag or an HTTP X-Robots-Tag header — and critically, the page must remain crawlable so Google can actually see that noindex instruction. Blocking a page in robots.txt while also adding noindex is self-defeating: Google can never read the noindex because you told it not to fetch the page.

If you’re new to how crawling, indexing, and ranking fit together, our explainer on how Google Search actually works covers the full pipeline.

Core Syntax: The Building Blocks

A robots.txt file is made of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules that apply to those agents.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/sitemap_index.xml

Key syntax rules that matter in practice:

• User-agent matching is longest-match wins. If Googlebot finds a group for User-agent: Googlebot, it obeys only that group and ignores the User-agent: * group entirely. This is the source of countless accidents: site owners add a Googlebot-specific group for one rule and unknowingly release Googlebot from every rule in the wildcard group.
• Rules are prefix-based. Disallow: /private blocks /private, /private/, /private-files/, and /private.html. If you only want to block the directory, write Disallow: /private/ with the trailing slash.
• The most specific rule wins. When Allow and Disallow rules conflict, Google applies the rule with the longest matching path. Allow: /blog/best-post/ beats Disallow: /blog/.
• An empty Disallow means allow everything. Disallow: with nothing after it permits full crawling. A single slash — Disallow: / — blocks the entire site. The difference between an SEO non-event and a catastrophe is one character.

Wildcards and Pattern Matching

Google, Bing, and most major crawlers support two pattern characters beyond simple prefixes:

• * matches any sequence of characters. Disallow: /*?sort= blocks any URL containing the sort parameter anywhere in its path or query string.
• $ anchors a rule to the end of the URL. Disallow: /*.pdf$ blocks URLs that end in .pdf, but not /file.pdf?download=true.

Practical patterns worth stealing:

# Block internal site search results
Disallow: /search/
Disallow: /*?s=

# Block faceted navigation parameters
Disallow: /*?*filter=
Disallow: /*?*price_min=

# Block add-to-cart and checkout actions
Disallow: /*?add-to-cart=
Disallow: /cart/
Disallow: /checkout/

These patterns matter most on e-commerce sites, where parameter combinations can generate millions of near-duplicate crawlable URLs. That waste has a real cost, which we explain in depth in our companion article on crawl budget optimization.

The Seven Most Common robots.txt Mistakes

1. Blocking CSS and JavaScript

A relic of pre-2015 SEO advice. Google renders pages like a browser; if it cannot fetch your CSS and JS, it may see a broken page, misjudge mobile-friendliness, and fail to see content loaded client-side. Never disallow /wp-includes/, /assets/, or theme directories wholesale.

2. The leftover staging block

Developers rightly add Disallow: / to staging environments — and then migrate that file to production during launch. Every technical SEO has seen a site invisible in search for weeks because of this. Add a robots.txt check to your launch checklist, and monitor it: Google Search Console’s Settings → robots.txt report shows the file Google last fetched.

3. Using robots.txt to “remove” pages from Google

As explained above, disallowing a URL does not deindex it. For removal, use noindex, a 404/410 status, or the URL Removal tool — and keep the URL crawlable until Google processes the change.

4. Case-sensitivity blindness

Paths in robots.txt are case-sensitive. Disallow: /Admin/ does not block /admin/. Match the exact casing your URLs actually use.

5. Blocking resources needed by rendered pages

Blocking an API endpoint like /api/ feels safe until you remember your JavaScript framework fetches product data from it. Googlebot’s renderer will then see empty product grids. Test important pages with the URL Inspection tool’s live test after any robots.txt change.

6. Contradictory agent groups

Because specific user-agent groups override the wildcard group completely, a file like this is a trap:

User-agent: *
Disallow: /private/

User-agent: Googlebot
Disallow: /tmp/

Googlebot here is allowed into /private/ because it only reads its own group. If you use specific groups, repeat every rule that should apply.

7. Forgetting the Sitemap directive

The Sitemap: line is the one directive that takes a full absolute URL and can appear anywhere in the file. It is a free, standards-compliant way to tell every crawler where your sitemap lives — including crawlers you never registered with in any webmaster console.

Advanced Directives and Modern Extensions

Crawl-delay

Google ignores Crawl-delay entirely (crawl rate for Googlebot is managed automatically). Bing and Yandex honor it. If server load is your concern, a value of Crawl-delay: 5 for aggressive secondary bots is reasonable, but fixing server performance is the better long-term answer — see our guide to HTTPS, speed, and security basics.

Managing AI crawlers

The biggest robots.txt shift of the last three years is the arrival of AI user agents: GPTBot, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended, CCBot, Bytespider, and others. Each can be allowed or blocked independently:

# Allow AI answer engines to cite you
User-agent: PerplexityBot
Allow: /

# Opt out of Google AI training while keeping normal Google Search
User-agent: Google-Extended
Disallow: /

Note that Google-Extended controls model training use, not whether you appear in Google Search or AI Overviews. Whether to block AI crawlers is a genuine strategic decision — blocking them protects content from training use but can remove you from AI-generated answers, which are becoming a real discovery channel. We break down that trade-off in our guides to GEO vs SEO strategy and getting cited in AI Overviews. Alongside robots.txt, an emerging convention worth adopting is the llms.txt file — you can build one in seconds with our free llms.txt generator.

Size and format limits

Google enforces a 500 KiB limit on robots.txt; rules beyond that are ignored. The file must be UTF-8 encoded plain text served with a 200 status. A 4xx response makes Google treat the site as fully crawlable; a persistent 5xx response can make Google throttle or stop crawling entirely — one more reason to monitor the file’s availability, not just its content.

A Sensible Default for WordPress Sites

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /*?add-to-cart=
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

Sitemap: https://yourdomain.com/sitemap_index.xml

Short, permissive, and boring — exactly what a robots.txt should be. Most sites need fewer rules than their owners think. Every Disallow line is a small bet that you understand your URL structure perfectly; keep the number of bets low.

How to Test Changes Safely

1. Draft the change locally and diff it against the current file.
2. Validate the syntax (Search Console’s robots.txt report flags parse errors line by line).
3. Run a live URL Inspection on two or three important pages to confirm they remain crawlable and render correctly.
4. After deployment, watch the Crawl Stats report for a week. A sudden drop in crawled URLs or a spike in “Blocked by robots.txt” in the Pages report is your early warning.

Frequently Asked Questions

Does every site need a robots.txt file?

Technically no — a missing file (404) is treated as “crawl everything.” But a minimal file with a Sitemap directive costs nothing and prevents your server logs from filling with 404s for the file. Every production site should have one.

Can robots.txt hurt my rankings?

Indirectly, yes, and badly. Blocking rendering resources, blocking important sections, or serving 5xx errors on the file itself can suppress crawling and indexing across the whole site. The file cannot boost rankings, but it can absolutely destroy them.

Do bad bots obey robots.txt?

No. robots.txt is a request, not access control. Scrapers and malicious bots ignore it. Anything genuinely private needs authentication, IP restrictions, or a firewall — never rely on robots.txt for security, and remember the file is public, so it can even advertise the paths you’d rather hide.

How often does Google re-fetch robots.txt?

Google caches it for up to 24 hours in normal conditions. Urgent fixes propagate within a day; you can also request a recrawl via the Search Console robots.txt report.

Next in this series: once your crawl rules are clean, make sure crawlers can discover everything you do want indexed — start with our comparison of XML and HTML sitemaps, and our beginner’s guide to site architecture and internal linking.