Should you block AI crawlers in robots.txt?
Blocking a training crawler keeps your text out of a future model. Blocking a retrieval crawler removes you from answers being given today. Most companies want the second and think they are choosing the first, because the names look alike.
This question is usually asked as a values question and answered as a technical one, which is why so many sites end up with a robots file that does the opposite of what the business wanted.
Two kinds of crawler, two kinds of consequence
| Purpose | What blocking it does | What it costs you |
|---|---|---|
| Training | Your pages are less likely to be used as material for a future model | Nothing this quarter. Over years, a weaker general sense of who you are |
| Live retrieval | The assistant cannot fetch your page while answering a question right now | Immediate absence from answers about your own subject |
Providers publish which of their user agents does which, and the mapping is not the same across providers. One vendor separates them cleanly; another uses the same agent for both; a third has a directive that governs use in generative answers without governing the crawl at all. Read each provider's own documentation, because a blocklist copied from a blog post is almost always a mix of names from three different eras.
Copying somebody else's robots.txt is how sites disappear from their own subject.
A decision that fits most companies
Allow both. Your problem is obscurity, not extraction. A model that has read your work and an assistant that can quote it are both routes to being considered.
Publishing, courses, research, paid archives. Block training crawlers, allow retrieval crawlers where it makes sense, and pursue licensing separately. The two are not in conflict.
The robots file is the wrong instrument entirely. It is a request, honoured voluntarily. Anything that genuinely must not be read needs authentication.
What a workable robots file looks like
- Deny nothing by default on your public content; deny your application paths, your API and your authentication routes.
- Name individual crawlers explicitly rather than using a blanket rule, so that a future rename does not silently change your position.
- Keep a comment beside every deny line saying why it is there and when it was added. Robots files outlive the people who wrote them.
- Point at a sitemap that is genuinely current.
- Re-read it every six months, because the agent names change more often than anything else in this field.
The argument for being read
For most businesses the realistic risk is not that a model learns what you know. It is that a model learns what your competitor knows, describes the subject in their terms, and never has cause to mention you. Absence is not neutrality. It is a decision to let somebody else answer the questions your market is asking.
Questions this settles
No. A robots directive affects future crawling. Material already used in a released model is not withdrawn by changing a file.
Blocking a provider's separate AI directive should not affect conventional search crawling, but the two are configured independently and it is easy to block the wrong agent. Verify with a live fetch rather than assuming.
The major named ones publish that they do. It remains a voluntary convention, so it is a policy signal rather than a control.