Robots.txt Generator & Tester
Build a valid robots.txt file by adding allow/disallow rules per user-agent — then verify whether any URL is allowed or blocked for a specific crawler.
Fetch a live robots.txt from any domain, or paste one you already have. Then enter URLs to see whether each is Allowed or Blocked for your chosen crawler.
| URL / Path | Verdict | Matched directive |
|---|
Frequently Asked Questions
robots.txt file lives at the root of your domain (e.g., https://example.com/robots.txt) and tells crawlers which pages they are allowed to access. It follows the Robots Exclusion Standard. Note: it is advisory, not a security mechanism — malicious bots can and do ignore it.noindex meta tag instead.User-agent: * applies rules to all crawlers. You can target specific crawlers by name — for example, User-agent: Googlebot for Google's main crawler or User-agent: Bingbot for Microsoft Bing. Specific rules take precedence over wildcard rules for that bot.Sitemap: directive in robots.txt makes it easy for crawlers to discover your XML sitemap without it needing to be submitted via Google Search Console. You can include multiple Sitemap: lines for multiple sitemaps.<meta name="robots">) controls whether a page can be indexed or its links followed — it operates at the indexing level. A page blocked in robots.txt cannot be crawled to read the meta robots tag, so you cannot use meta robots noindex on a page blocked by robots.txt.Sitemap: directive followed by the full absolute URL of your sitemap file, for example: Sitemap: https://example.com/sitemap.xml. This directive applies to all bots regardless of which User-agent block it appears in, and you can list multiple sitemaps. Most SEO professionals place the Sitemap directive at the very end of the robots.txt file for readability.About This Robots.txt Generator & Tester
This free tool lets you both build and verify a robots.txt file. Use the Generator tab to create allow/disallow rules for any user-agent, then switch to the Tester tab to confirm whether specific URLs are allowed or blocked — by fetching a live site or pasting existing content.
A correctly formatted robots.txt file tells search engine crawlers which parts of your site to index and which to skip. Syntax errors in robots.txt can inadvertently block your entire site from being crawled.
When to use this tool
- Blocking admin, login, and private pages from being crawled
- Allowing Googlebot while blocking specific other bots
- Adding a sitemap directive to help crawlers discover content
- Verifying that specific URLs are allowed or blocked before deploying
Standards & References
How It Works
Add Rules
Select a user-agent and choose Allow or Disallow for a specific path. Add as many rules as needed, or use a quick preset to start.
Preview Instantly
The robots.txt output updates in real time as you add or modify rules. Rules are grouped by user-agent in the correct format.
Test & Verify
Switch to the Tester tab to fetch or paste a robots.txt and verify whether any URL is allowed or blocked for your chosen crawler.
Common Use Cases
Block Admin Areas
Disallow /admin/, /wp-admin/, and /login/ paths to prevent crawlers from wasting crawl budget on private areas.
Block Staging Sites
Use Disallow: / for all bots on staging environments to prevent duplicate content issues if the staging URL is ever discovered.
Crawl Budget Management
Block paginated pages, filter/sort URLs, and internal search result pages to focus Google's crawl budget on your most important content.
Control Media Indexing
Use Googlebot-Image disallow rules to prevent certain image directories from appearing in Google Image search results.
Block Tag & Category Pages
On blog and CMS sites, disallow tag, category, and archive pages that create shallow content which dilutes the authority of core pages.
Sitemap Discovery
Include the Sitemap: directive to help all search engines discover your XML sitemap automatically without manual submission in each webmaster tool.
Related SEO Tools
Related Articles
View all articles
Testing Robots.txt Is More Valuable Than Generating It — The URL Matching Rules That Surprise Everyone
Testing whether a URL is allowed or blocked by your robots.txt reveals more than generating new rules — most robots.txt errors come from not knowing which rules match which URLs. Here's the prefix-matching logic that catches more paths than expected (/admin matches /administration), why query strings can't be targeted by robots.txt path rules, the seven URL variants to test for every rule, and how Allow + Disallow specificity interaction produces counterintuitive results.
Robots.txt Can Only Ask — It Can't Enforce, Hide, or Remove From Google's Index
Robots.txt can only ask crawlers to stay away — it can't enforce it, it can't hide content, and it cannot remove an already-indexed URL from Google's results. Here's why Disallow prevents crawling but not indexing (and why disallowing a page you want noindexed is self-defeating), why robots.txt paths are public information that scrapers deliberately read, and the correct tool for each goal: Disallow for crawl budget, noindex for search results, authentication for actual privacy.
Robots.txt, Meta Robots, X-Robots-Tag: Which One Do You Actually Need? A Goal-First Framework
Robots.txt, meta robots, and X-Robots-Tag aren't competing options — each addresses a different goal (crawl budget, index exclusion for HTML, index exclusion for PDFs/files), and "belt and suspenders" combining robots.txt blocking with noindex doesn't add safety, it disables the noindex entirely. Here's a goal-first decision framework for which mechanism to reach for, and why genuinely sensitive content needs authentication, not extra robots directives.
AI Crawlers and the New robots.txt Reality: GPTBot, Google-Extended, and ClaudeBot
GPTBot, ClaudeBot, Google-Extended, and a growing list of AI training crawlers now require active robots.txt management. Here's every major AI crawler and its user agent, how to block them selectively, the distinction between blocking Google-Extended vs Googlebot, and what "respect robots.txt" actually means in practice.
Robots.txt Mistakes That Silently Kill SEO — and the Correct Configurations
A wrong robots.txt can deindex your entire site and nobody warns you until rankings collapse. Here's the most dangerous mistakes (Disallow: /, blocking CSS/JS), why robots.txt can't prevent indexation alone, and correct configurations for common scenarios.
