How to manage AI bots in robots.txt
7 min read · Updated: 20 Sep 2026
Contents
Start by checking which bots visit
In your server logs you will see identities such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Some collect content for model training, others find pages to cite in answers.
Before deciding, sort out which bot comes for which purpose. The same company may use different bots for training and for search.
Separate search bots from training bots
If you want to appear as a source in answers, you need to keep search bots allowed. Whether to block training bots is a separate choice about your content being used to train models.
For example, blocking GPTBot does not by itself stop you from being cited in ChatGPT search as long as OAI-SearchBot is allowed.
Write the rule correctly
Open a separate "User-agent" block for each bot and list the paths you allow or block under it. A restriction in the general "User-agent: *" block affects every bot you have not written a rule for.
After the change, open the file in a browser and check that it is really live. A cache or CDN may still serve the old file.
Do not forget the firewall
Even when robots.txt allows a bot, a firewall or bot protection can block it. The bot then never sees the page and the permission in the file stays on paper.
AILens's AGT-012 control checks whether a bot allowed by robots.txt can really reach the page.
Write your decision down
Keep a short note on which bot you allowed or blocked and why. It makes client reporting and the next change easier.
Bot lists change. At least once a year, check the companies' current documentation and review your rules.
Check what this article covers on your own site.
Check your site