Robots.txt Generator — Create Crawler Access Rules for Any Website
Learn how robots.txt works, the difference between disallowing crawling vs. preventing indexing, common mistakes to avoid, and how to generate a correct robots.txt file for any site with a free tool.
By sadiqbd · June 6, 2026
robots.txt is a small file with significant power over how your site gets crawled
Every website's crawl experience starts with a single text file at the root: robots.txt. Search engine bots check this file before crawling anything else, and it sets the rules — which pages they can visit, which they can't, and where to find the sitemap. Get it right and you direct crawl budget efficiently. Get it wrong and you can accidentally block your entire site from Google.
A robots.txt generator builds a correctly formatted file based on your settings — with no risk of syntax errors that could have unintended consequences.
What robots.txt Does
The robots.txt file lives at https://yourdomain.com/robots.txt and follows the Robots Exclusion Protocol. It's a text file with a specific syntax that tells search engine crawlers (user-agents) what they're allowed to access on your site.
Key points:
- It's a directive, not a lock — robots.txt requests polite compliance. Malicious bots ignore it. Legitimate search engines (Googlebot, Bingbot, etc.) respect it.
- Blocking in robots.txt prevents crawling, not indexing. If another site links to a blocked URL, Google might still index it (they'll just have less information about it). To prevent indexing, use
noindexmeta robots tags instead. - Syntax errors in robots.txt can block more than intended — a single typo in a path can disallow entire sections of your site.
robots.txt Syntax
A robots.txt file is made up of rule sets, each applying to one or more user-agents:
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /
User-agent: Googlebot
Disallow: /staging/
Allow: /
Sitemap: https://example.com/sitemap.xml
Key directives
User-agent: — Specifies which bot(s) the following rules apply to.
*— all botsGooglebot— Google's crawlerBingbot— Bing's crawlerGPTBot— OpenAI's crawleranthropic-ai— Anthropic's crawler- Any specific bot name
Disallow: — Paths the user-agent should not access.
Disallow: /admin/— block all URLs starting with/admin/Disallow: /— block the entire site (common accident — be careful)Disallow:— empty disallow = allow everything (equivalent to no rule)
Allow: — Explicitly allow a path, even within a disallowed parent.
Disallow: /images/+Allow: /images/product/— blocks all images except product photos- More specific rules take precedence over less specific ones
Sitemap: — URL of the XML sitemap. Helps search engines discover all your pages.
Crawl-delay: — Requests a delay (in seconds) between requests. Respects server load. Use if your server struggles with bot traffic.
How to Use the robots.txt Generator on sadiqbd.com
- Select which bots to configure — all bots (
*), or specific crawlers - Enter paths to disallow — directories or specific URLs to block
- Enter paths to allow (if needed within disallowed areas)
- Enter your sitemap URL — for search engine discovery
- Generate — the tool produces correctly formatted robots.txt content
- Upload to your domain root as
robots.txt
Real-World Examples
Standard website robots.txt
Allow all crawlers access to everything except admin and private areas, with sitemap reference:
User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /user/dashboard/
Disallow: /checkout/
Disallow: /cart/
Allow: /
Sitemap: https://example.com/sitemap.xml
E-commerce site with faceted navigation
Faceted navigation (filtering by colour, size, price) creates thousands of near-duplicate URLs with parameters:
User-agent: *
Disallow: /search?
Disallow: /?color=
Disallow: /?size=
Disallow: /?sort=
Disallow: /products?
Allow: /products/
Allow: /
Sitemap: https://shop.example.com/sitemap.xml
This prevents crawl budget waste on filter combinations while keeping actual product pages accessible.
Blocking AI training crawlers
Some site owners want to block AI company crawlers while allowing search engines:
User-agent: GPTBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xml
Specific crawlers are blocked entirely while Googlebot and Bingbot can access everything except the admin area.
Staging environment block
User-agent: *
Disallow: /
A staging site should be completely blocked. This single-line disallow prevents all bots from indexing staging content. Combine with password protection for full security.
Allowing access within a blocked area
You block most of /assets/ but want Googlebot to access product images for image search:
User-agent: Googlebot
Allow: /assets/products/
Disallow: /assets/
User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xml
More specific Allow rule overrides the broader Disallow.
Common robots.txt Mistakes
Disallow: / in production. The most catastrophic error — blocks all bots from your entire site. If you see this in your production robots.txt, remove it immediately. Sometimes accidentally deployed from a staging environment.
Blocking CSS and JavaScript files. Old guidance recommended blocking these; modern guidance is the opposite. Googlebot renders pages like a browser — blocking CSS and JS prevents it from seeing the page as users do, potentially misunderstanding your content or page experience.
No sitemap directive. Not required but highly recommended. The Sitemap: directive helps search engines find all your pages without relying solely on crawl discovery.
Disallowing URL parameters instead of using canonical tags. Blocking /page?param=value in robots.txt prevents crawling but not indexing (if linked elsewhere). Use canonical tags for duplicate parameter URL management alongside or instead of robots.txt.
Over-blocking. Blocking too much wastes the potential of pages that should be indexed. Audit your robots.txt regularly to ensure Disallow rules are intentional.
robots.txt vs. Meta Robots Tag
| robots.txt | Meta Robots Tag | |
|---|---|---|
| Prevents crawling | Yes | No |
| Prevents indexing | No | Yes |
| Page-level control | No | Yes |
| Whole-section control | Yes | No |
| Works if page isn't crawled | N/A | No (tag isn't seen) |
Use robots.txt to prevent crawling (URL parameter pages, admin sections, duplicate content sources).
Use meta robots noindex to prevent indexing (thank-you pages, legal pages, internal search results you want crawlable but not indexed).
For pages that must not be indexed: use both — robots.txt to reduce crawl, meta robots noindex to prevent indexing of any version that gets crawled anyway (via external links, for example).
Verifying robots.txt
- Direct URL: Visit
https://yourdomain.com/robots.txtin a browser — it should render as plain text - Google Search Console: Settings → robots.txt viewer shows what Google sees and whether it can access the file
- Google Search Console URL Inspection: Test whether a specific URL is blocked by robots.txt
- robots.txt tester tools: Test specific paths against your rules to verify Allow/Disallow behaviour
Frequently Asked Questions
Does robots.txt prevent pages from being indexed?
No — it prevents crawling. If a blocked page is linked from other sites, Google may still index it with limited information (just the URL and anchor text from incoming links). Use noindex meta robots tag to prevent indexing.
Should I disallow crawlers from my paginated pages? Not as a general rule. Paginated pages (page 2, page 3) have some indexed value. If you want to consolidate signals, use canonical tags on pagination rather than robots.txt blocking.
Can I have multiple Sitemap: directives?
Yes — you can include multiple Sitemap: lines for separate sitemaps (news sitemap, image sitemap, video sitemap).
What if my robots.txt has a syntax error? Googlebot is generally forgiving of minor syntax issues and interprets ambiguous rules conservatively. However, explicitly wrong rules (missing trailing slash, wrong path) can cause unintended blocking. Use the generator to avoid syntax issues.
Is the robots.txt generator free? Yes — completely free, no sign-up required.
robots.txt is deceptively simple — a small plain text file with a few rules. But the consequences of getting it wrong range from harmless (slightly inefficient crawl budget) to catastrophic (entire site deindexed). The generator gets the syntax right so the only decisions required are strategic ones.
Try the robots.txt Generator free at sadiqbd.com — generate a correctly formatted robots.txt for any site configuration instantly.
