Robots.txt Generator

Build a valid robots.txt file by adding allow/disallow rules per user-agent. Control which pages and bots can crawl your website.

Sitemap & Crawl Delay
Rules
robots.txt Output 

            

          

        

      


      

        
Frequently Asked Questions

        

          

            

            

              
The robots.txt file lives at the root of your domain (e.g., https://example.com/robots.txt) and tells crawlers which pages they are allowed to access. It follows the Robots Exclusion Standard. Note: it is advisory, not a security mechanism — malicious bots can and do ignore it.

            

          

          

            

            

              
No. A Disallow rule prevents Googlebot from crawling the page, but if the page has external links pointing to it, Google may still show it in search results with a "No information is available for this page" snippet. To remove a page from Google's index, use a noindex meta tag instead.

            

          

          

            

            

              
User-agent: * applies rules to all crawlers. You can target specific crawlers by name — for example, User-agent: Googlebot for Google's main crawler or User-agent: Bingbot for Microsoft Bing. Specific rules take precedence over wildcard rules for that bot.

            

          

          

            

            

              
Yes, it is recommended. The Sitemap: directive in robots.txt makes it easy for crawlers to discover your XML sitemap without it needing to be submitted via Google Search Console. You can include multiple Sitemap: lines for multiple sitemaps.

            

          

        

      


      
      

        
How It Works

        

          

            

            
Add Rules

            
Select a user-agent and choose Allow or Disallow for a specific path. Add as many rules as needed, or use a quick preset to start.

          

          

            

            
Preview Instantly

            
The robots.txt output updates in real time as you add or modify rules. Rules are grouped by user-agent in the correct format.

          

          

            

            
Copy or Download

            
Copy the generated content or download it as robots.txt and upload it to the root directory of your web server.

          

        

      


      
      

        
Common Use Cases

        

          

            
Block Admin Areas

            
Disallow /admin/, /wp-admin/, and /login/ paths to prevent crawlers from wasting crawl budget on private areas.

          

          

            
Block Staging Sites

            
Use Disallow: / for all bots on staging environments to prevent duplicate content issues if the staging URL is ever discovered.

          

          

            
Crawl Budget Management

            
Block paginated pages, filter/sort URLs, and internal search result pages to focus Google's crawl budget on your most important content.

          

          

            
Control Media Indexing

            
Use Googlebot-Image disallow rules to prevent certain image directories from appearing in Google Image search results.

          

          

            
Block Tag & Category Pages

            
On blog and CMS sites, disallow tag, category, and archive pages that create shallow content which dilutes the authority of core pages.

          

          

            
Sitemap Discovery

            
Include the Sitemap: directive to help all search engines discover your XML sitemap automatically without manual submission in each webmaster tool.

          

        

      


      
      

        
Related SEO Tools

        

          

            
XML Sitemap Generator

          

          

            
Meta Tag Generator

          

          

            
Canonical Tag Generator

          

          

            
SEO Checklist