michael yost.
← All tools
04 / Free marketing tools

AI crawler rules.

Decide which AI crawlers you want to invite in. Set your preferences, then copy the matching robots.txt rules.

CrawlerAllow access

robots.txt is a request that cooperative bots honor, not a security control. Some AI agents do not identify themselves at all, and only server or firewall rules can stop those.

OpenAI · Anthropic · Google · Perplexity · Apple · Meta · Common Crawl

robots.txt 

          

            Plain text / UTF-8
          

        

      

      

        
How it works

        
    
          
  1. 
            01
            
    
              
    Choose the crawlers
    
              
    
                Start from a preset or set each bot. Blocked bots share one Disallow group. Allowed bots need no
                group of their own, so they keep following your existing rules.
              
    
            
    
          
    2. 
          
  2. 
            02
            
    
              
    Merge with your existing rules
    
              
    
                Check for existing groups for these bots. Do not remove unrelated rules or sitemap
                declarations.
              
    
            
    
          
    3. 
          
  3. 
            03
            
    
              
    Publish at the root
    
              
    
                Add the rules to /robots.txt. Bots choose whether to honor them, and changes may take time to
                be noticed.
              
    
            
    
          
    4.