Robots.txt Generator & Tester

Build a valid robots.txt file by adding allow/disallow rules per user-agent — then verify whether any URL is allowed or blocked for a specific crawler.

Sitemap & Crawl Delay
Rules
robots.txt Output 

    

  









  
Frequently Asked Questions

  

    
    
The robots.txt file lives at the root of your domain and tells crawlers which pages they are allowed to access. It follows the Robots Exclusion Standard. Note: it is advisory, not a security mechanism — malicious bots can and do ignore it.

  

  

    
    
No. A Disallow rule prevents Googlebot from crawling the page, but if the page has external links pointing to it, Google may still show it in search results with a "No information is available for this page" snippet. To remove a page from Google's index, use a noindex meta tag instead.

  

  

    
    
User-agent: * applies rules to all crawlers. You can target specific crawlers by name — for example, User-agent: Googlebot for Google's main crawler or User-agent: Bingbot for Microsoft Bing. Specific rules take precedence over wildcard rules for that bot.

  

  

    
    
Yes, it is recommended. The Sitemap: directive in robots.txt makes it easy for crawlers to discover your XML sitemap without it needing to be submitted via Google Search Console. You can include multiple Sitemap: lines for multiple sitemaps.

  

  

    
    
robots.txt controls whether a crawler can access and fetch a page at all — it operates at the crawl level. The meta robots tag controls whether a page can be indexed or its links followed — it operates at the indexing level. A page blocked in robots.txt cannot be crawled to read the meta robots tag, so you cannot use meta robots noindex on a page blocked by robots.txt.

  

  

    
    
Google does not support the Crawl-delay directive in robots.txt. To slow down Googlebot, use the crawl rate settings in Google Search Console. Bing and some other crawlers do honour Crawl-delay, so it is still worth including for them.

  






  
About This Robots.txt Generator & Tester

  
This free tool lets you both build and verify a robots.txt file. Use the Generator tab to create allow/disallow rules for any user-agent, then switch to the Tester tab to confirm whether specific URLs are allowed or blocked — by fetching a live site or pasting existing content.

  
When to use this tool

  
    
    
  • Blocking admin, login, and private pages from being crawled
    • 
    
  • Allowing Googlebot while blocking specific other bots
    • 
    
  • Adding a sitemap directive to help crawlers discover content
    • 
    
  • Verifying that specific URLs are allowed or blocked before deploying
    • 
  

  
Standards & References

  





  
Related SEO Tools

  

    XML Sitemap Generator
    Meta Tag Generator
    Canonical Tag Generator
    SEO Checklist
  







          

        
                        

          

            

              

                
Related Articles

                
In-depth guides and technical articles.

              

              View all →
            

            

                            

                
                                    
                                  
                

                  

                                        Seo
                                        Jul 21, 2026                  

                  Robots.txt Was Informal for 28 Years — What RFC 9309 Actually Standardized and What It Still Leaves Open
                                    
Robots.txt was an informal convention for 28 years before the IETF formalized it as RFC 9309 in September 2022 — and the standard still doesn't cover crawl-delay (Bing respects it, Google never has) or settle whether crawlers are legally required to comply. Here's what RFC 9309 actually specifies (500 KiB max size, longest-match-wins, caching rules), how the standard naturally accommodates the new wave of AI training crawlers via multiple User-agent blocks, and the legal landscape's continued ambiguity post-standardization.

                                  

              

                            

                
                                    
                                  
                

                  

                                        Seo
                                        Jun 25, 2026                  

                  Testing Robots.txt Is More Valuable Than Generating It — The URL Matching Rules That Surprise Everyone
                                    
Testing whether a URL is allowed or blocked by your robots.txt reveals more than generating new rules — most robots.txt errors come from not knowing which rules match which URLs. Here's the prefix-matching logic that catches more paths than expected (/admin matches /administration), why query strings can't be targeted by robots.txt path rules, the seven URL variants to test for every rule, and how Allow + Disallow specificity interaction produces counterintuitive results.

                                  

              

                            

                
                                    
                                  
                

                  

                                        Seo
                                        Jun 21, 2026                  

                  Robots.txt Can Only Ask — It Can't Enforce, Hide, or Remove From Google's Index
                                    
Robots.txt can only ask crawlers to stay away — it can't enforce it, it can't hide content, and it cannot remove an already-indexed URL from Google's results. Here's why Disallow prevents crawling but not indexing (and why disallowing a page you want noindexed is self-defeating), why robots.txt paths are public information that scrapers deliberately read, and the correct tool for each goal: Disallow for crawl budget, noindex for search results, authentication for actual privacy.

                                  

              

                            

                
                                    
                                  
                

                  

                                        Seo
                                        Jun 14, 2026                  

                  Robots.txt, Meta Robots, X-Robots-Tag: Which One Do You Actually Need? A Goal-First Framework
                                    
Robots.txt, meta robots, and X-Robots-Tag aren't competing options — each addresses a different goal (crawl budget, index exclusion for HTML, index exclusion for PDFs/files), and "belt and suspenders" combining robots.txt blocking with noindex doesn't add safety, it disables the noindex entirely. Here's a goal-first decision framework for which mechanism to reach for, and why genuinely sensitive content needs authentication, not extra robots directives.

                                  

              

                            

                
                                    
                                  
                

                  

                                        Seo
                                        Jun 10, 2026                  

                  AI Crawlers and the New robots.txt Reality: GPTBot, Google-Extended, and ClaudeBot
                                    
GPTBot, ClaudeBot, Google-Extended, and a growing list of AI training crawlers now require active robots.txt management. Here's every major AI crawler and its user agent, how to block them selectively, the distinction between blocking Google-Extended vs Googlebot, and what "respect robots.txt" actually means in practice.