Robots.txt Syntax Explained: Directives, Wildcards, and the Mistakes That Cost Traffic

Robots.txt is a plain-text file at your domain's root that tells well-behaved crawlers which parts of your site they may access. It's a simple format, but small syntax mistakes here have an outsized blast radius — a single misplaced character can deindex an entire site, which makes it one of the highest-stakes files in technical SEO relative to its size.

The Core Directives

User-agent: specifies which crawler a rule block applies to (* for all crawlers, or a specific bot name like Googlebot). Disallow: specifies a path pattern crawlers should not access. Allow: creates an exception within a disallowed path — useful when you want to block a whole directory except one specific file or subfolder inside it. Sitemap: points to your XML sitemap's location, and can appear anywhere in the file regardless of user-agent blocks.

Wildcards and Pattern Matching

The asterisk (*) matches any sequence of characters, letting you block patterns like Disallow: /*?sort= to catch every URL with a sort-order query parameter regardless of the base path. The dollar sign ($) anchors a match to the end of a URL — Disallow: /*.pdf$ blocks PDF files specifically without also catching a URL that merely contains ".pdf" as a substring elsewhere in the path.

The Catastrophic Mistake: Blocking Everything

A stray Disallow: / under User-agent: * blocks crawler access to the entire site. This happens most often when a staging or development robots.txt file — intentionally blocking everything during development — gets deployed to production by accident. Always check your live robots.txt after any deployment, and consider adding an automated monitoring check (many uptime tools support this) that alerts you if the file's content changes unexpectedly.

Robots.txt Blocks Crawling, Not Indexing

This is the single most misunderstood point about robots.txt: disallowing a URL prevents Googlebot from crawling it, but if other sites link to that URL, Google can still index the bare URL (with no description, since it can't crawl the content) and show it in results. To truly keep a page out of the index, use a noindex meta tag or HTTP header instead — and critically, don't block that page in robots.txt at the same time, because a blocked page can never be crawled to see the noindex tag in the first place, which defeats the purpose entirely.

Common Real-World Mistakes to Check For

• Blocking CSS or JavaScript files needed for rendering, which can cause Google to see a broken or incomplete version of your page during rendering.
• Case-sensitivity errors — robots.txt paths are case-sensitive, so /Blog/ and /blog/ are treated as entirely different paths.
• Leftover rules from a site migration blocking an old URL structure that no longer needs special handling.
• Conflicting Allow/Disallow rules where crawler behavior depends on rule specificity, which isn't always intuitive without testing.

Test Before You Deploy

Use Google Search Console's robots.txt Tester (or the open-source robots.txt parser Google publishes) to validate that specific URLs are treated as expected before pushing a change live. Testing takes two minutes and prevents the kind of mistake that can take weeks to fully recover from once Google has deindexed pages based on a bad rule. For a deeper look at this, see our guide, Tracking & Analytics.

Remember That Other Bots Read Robots.txt Too

Beyond search engines, a wide range of crawlers — SEO tools like Ahrefs and Semrush, AI training crawlers, and various scrapers — respect (or claim to respect) robots.txt directives. You can use specific user-agent blocks to manage these separately from search engines if desired, for instance restricting AI-specific crawlers like GPTBot without affecting Googlebot's access at all. Keep in mind robots.txt is a voluntary honor system, not an access control mechanism — a malicious scraper can simply ignore it, so never rely on it to protect genuinely sensitive content. For a deeper look at this, see our guide, Title Tags, Meta Descriptions & Headers.