Robots.txt Was Informal for 28 Years — What RFC 9309 Actually Standardized and What It Still Leaves Open

robots.txt sits at a strange intersection of web standards — it's been an informal convention since 1994, formally standardized by the IETF only in 2022 (RFC 9309), and the gap between "what crawlers actually do" and "what the standard says they should do" explains many of the inconsistencies SEO practitioners encounter

The previous articles on this site covered robots.txt mistakes, AI crawler directives, the robots.txt vs meta robots decision framework, why robots.txt can't enforce removal, and URL matching rule testing. This article addresses the robots.txt standard itself — its formalization history, what RFC 9309 actually specifies, and the specific areas where crawler behaviour still diverges from the standard or from each other.

The robots.txt history: 28 years as an informal convention

The Robots Exclusion Protocol was created in 1994 by Martijn Koster, in response to early web crawlers causing server overload. It started as a simple, voluntary convention — a text file at a well-known location that crawlers could check before crawling.

For 28 years, there was no formal standard — only the original informal specification and de facto conventions that emerged from how major search engines (initially AltaVista, Lycos, and others; later Google, Bing) implemented it. Different crawlers interpreted edge cases differently because there was no authoritative specification to resolve disputes.

RFC 9309 (September 2022): the IETF formally standardized the Robots Exclusion Protocol, led significantly by Google's proposal and years of advocacy. This finally created an authoritative specification for behaviour that had previously been governed only by convention and the dominant search engines' documented behaviour.

What RFC 9309 actually specifies (and what it leaves open)

RFC 9309 standardizes:

File location and access: robots.txt must be at the root of the domain (https://example.com/robots.txt), accessible via HTTP/HTTPS, and the response must be a text/plain content type (though crawlers are advised to be lenient about content-type).

Maximum file size: crawlers should process at least the first 500 kibibytes of a robots.txt file — files larger than this may be truncated, with only the rules within that limit being honoured.

Caching: crawlers should cache robots.txt for up to 24 hours, but can use a cached version longer if the server is unreachable when attempting to refresh.

Allow and Disallow directive matching: the longest matching rule wins (not the first matching rule) — clarifying ambiguity that existed in informal specifications.

Unreachable robots.txt handling: if the robots.txt file returns a 5xx server error, crawlers should treat this as "fully disallow" (assume the worst) for some period, then retry. A 404 (not found) means "no restrictions" — the crawler can crawl freely.

RFC 9309 explicitly does NOT specify:

• The crawl-delay directive (used by some crawlers, ignored by Google, never standardized)
• The sitemap directive (widely supported as a de facto convention but not part of the formal RFC)
• Wildcard matching syntax details beyond basic * and $ (left somewhat to implementation)

Why Bing, Google, and other crawlers still differ

Even after RFC 9309's standardization, crawler behaviour isn't perfectly unified — because the RFC standardizes the core protocol while leaving implementation details to each crawler:

Crawl-delay: Bing and Yandex historically respected Crawl-delay: N (wait N seconds between requests). Google has never supported this directive — Google's crawl rate is instead managed through Search Console's crawl rate settings (largely automated) and the server's response time behaviour.

Case sensitivity in user-agent matching: RFC 9309 specifies user-agent matching should be case-insensitive, but earlier crawler implementations varied on this — older robots.txt files written with assumptions about case sensitivity may behave unexpectedly with crawlers that correctly follow the RFC's case-insensitive rule.

Wildcard interpretation depth: while * (any sequence) and $ (end of string) are standardized, more exotic patterns some crawlers historically supported (character classes, more complex regex-like syntax) are not part of RFC 9309 and may not be portable across crawlers.

The AI crawler proliferation since RFC 9309

Since the RFC's publication, a new category of crawler has proliferated: AI training data crawlers (GPTBot from OpenAI, Google-Extended, CCBot from Common Crawl, anthropic-ai, ClaudeBot, and others) that use the same robots.txt mechanism but for a different purpose — gathering training data rather than indexing for search results.

The RFC's framework accommodates this naturally: because robots.txt allows multiple User-agent blocks with different rules, site owners can permit search indexing crawlers while blocking AI training crawlers (or vice versa) using the existing standard:

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The trust dependency: RFC 9309, like the original 1994 convention, is fundamentally voluntary. A crawler that chooses to ignore robots.txt entirely faces no technical enforcement — only reputational consequences (being identified as a "bad bot" and subsequently IP-blocked) and, in some jurisdictions, potential legal consequences under computer access laws if circumventing robots.txt is found to violate terms of service.

Robots.txt and the legal landscape post-RFC

The formalization of RFC 9309 has had some legal significance in disputes about unauthorized web scraping:

The argument structure: in disputes where a company has scraped data from a website despite robots.txt disallowing it, the existence of a formal, IETF-standardized protocol strengthens the argument that the website owner had clearly communicated their access preferences through a recognized technical standard — relevant in cases invoking the Computer Fraud and Abuse Act (US) or similar computer misuse legislation in other jurisdictions.

The limitation: courts have generally been cautious about treating robots.txt violations alone as definitive evidence of unauthorized access, particularly for publicly accessible content. The legal weight of robots.txt remains contested and varies by jurisdiction — RFC 9309's standardization strengthens the technical argument but doesn't settle the legal question definitively.

How to use the Robots.txt Generator & Tester on sadiqbd.com

1. For RFC 9309 compliance: verify your robots.txt file is under 500 KiB, correctly placed at the domain root, and uses standard directive syntax that all RFC-compliant crawlers will interpret consistently
2. For multi-crawler policy: use separate User-agent blocks to differentiate search crawler permissions from AI training crawler permissions, as shown in the example above — this is the standard, RFC-compliant way to express different policies for different crawler purposes
3. For testing case sensitivity: verify your user-agent matching works correctly regardless of case — the RFC specifies case-insensitive matching, but testing against the actual crawler behaviour confirms your specific implementation handles this correctly

Frequently Asked Questions

Does RFC 9309 mean all crawlers are now legally required to respect robots.txt? No — RFC 9309 standardizes the technical protocol; it does not create a legal obligation for crawlers to comply. Robots.txt remains a voluntary convention from a legal compliance standpoint — there's no law that says "crawlers must obey robots.txt." What RFC 9309 does provide is an authoritative, unambiguous technical specification that removes prior excuses based on protocol ambiguity ("we interpreted the directive differently") and that may be referenced in legal disputes about access authorization, particularly in jurisdictions where unauthorized computer access is a relevant legal theory. Compliance remains driven by reputational incentives (search engines that don't respect robots.txt lose website operator trust and may face blocking) rather than legal mandate.

Is the Robots.txt Generator & Tester free? Yes — completely free, no sign-up required.

Try the Robots.txt Generator & Tester free at sadiqbd.com — build robots.txt rules and verify URL matching for any crawler.