robots.txt Generator

Build a valid, syntax-checked robots.txt with per-crawler rules, crawl-delay settings and sitemap declarations. All logic runs locally.

Directives

Generated robots.txt


      

      

        
      

    

    

      
Validation

      

    

  


  

  

    
What robots.txt Actually Does

    
robots.txt is a plain-text file served at the root of your domain (https://example.com/robots.txt) that tells well-behaved crawlers which paths they should or should not request. It is a convention, not a security mechanism — a malicious scraper will simply ignore it. Google and Bing both honour it.


    
Syntax Rules

    
    
      
  • Each rule is Field: value. Field names are case-insensitive but the value is not.
    • 
      
  • User-agent starts a new group; rules below apply to that crawler until the next User-agent.
    • 
      
  • Disallow: /path blocks a path. Allow: /path overrides a broader Disallow.
    • 
      
  • Sitemap: is a non-group directive — it can appear anywhere and applies site-wide.
    • 
      
  • Crawl-delay: is honoured by Bing and Yandex; Google ignores it.
    • 
      
  • Comments start with #.
    • 
    


    
What robots.txt Cannot Do

    
    
      
  • It cannot hide content — a Disallowed page can still be indexed if linked elsewhere, just without a snippet.
    • 
      
  • It does not remove a page from the index. Use noindex for removal.
    • 
      
  • It cannot protect sensitive files. Use authentication.
    • 
    


    
Common Mistakes

    
The worst mistake in SEO history is a live Disallow: / on a production site — it will deindex an entire domain within weeks. This tool never emits that pattern unless you explicitly add it, and it validates that your "Allow all" section is not accidentally overridden.

    
A second common error is blocking /wp-content/ on WordPress or /_next/static/ on Next.js — these contain CSS and JS that Google needs to render the page. Blocking them will hurt your Core Web Vitals scoring because Google can't fetch the resources it needs.

    
The third is Disallowing a URL and simultaneously applying noindex via meta tag. The crawler can't see the noindex because it is not allowed to fetch the page. If you need removal, use noindex without a Disallow.


    
Recommended Baseline

    
For 95% of sites, the safest baseline is:

    
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart

Sitemap: https://example.com/sitemap.xml

    
This keeps everything crawlable except cart and admin paths (which waste crawl budget), and gives the crawler an explicit sitemap URL.

  

















robots.txt FAQ




Does Disallow: / stop a page from appearing in Google?
No. It stops Google from fetching the page, but the URL can still appear in results with a title or snippet from links pointing at it. To remove a page from the index, use a noindex meta tag or HTTP header.


Can I block AI crawlers like GPTBot or CCBot?
Yes — add a separate User-agent: GPTBot group with Disallow: /. Note that some AI vendors do not honour robots.txt at all.


Does Google support Crawl-delay?
No. Google stopped honouring Crawl-delay in 2015. Use Google Search Console's crawl-rate setting instead. Bing and Yandex still support it.


Where does robots.txt go?
It must be at the root: https://example.com/robots.txt. A robots.txt at /blog/robots.txt applies only to /blog/.