What Is AI Bot Blocking?

AI bot blocking is the practice of adding rules to your website's robots.txt file that tell specific AI crawlers not to read your pages. For a health practice, the key point is that blocking a bot used for training is a different choice from blocking a bot that lets an AI tool show your practice to patients. Mixing them up can remove you from ChatGPT's search answers without protecting anything you cared about.

How AI bot blocking works

Your robots.txt file lives at yourpractice.com/robots.txt. Anyone can open it. It lists bot names, called user agents, and a rule under each. "Disallow: /" under a name tells that bot to stay out of the whole site. Google's documentation describes the file as a way to tell crawlers which URLs they can access, and says it is not a mechanism for keeping a page out of Google. It also says the instructions cannot enforce crawler behavior. Well-behaved crawlers honor them, and compliance is up to the crawler.

What makes AI blocking tricky is that every company splits its crawling into jobs. Here is the map.

• Training bots collect content that may be used to train models. OpenAI's GPTBot and Anthropic's ClaudeBot are in this group. Google-Extended is a token, not a bot, that controls use of content for Gemini training and grounding, and Google says it does not affect inclusion in Google Search.
• Search bots help an AI tool show websites in answers. OpenAI's OAI-SearchBot surfaces sites in ChatGPT search, and OpenAI says opted-out sites will not be shown in those answers, though they can still appear as navigational links. Anthropic's Claude-SearchBot indexes content to improve search result relevance, and Anthropic says blocking it may reduce your visibility and accuracy in user search results.
• User bots fetch a page because a person asked. OpenAI's ChatGPT-User and Anthropic's Claude-User are here. OpenAI says robots.txt rules may not apply to ChatGPT-User, since a user starts the visit.

OpenAI states that each of its settings is independent, and that it can take about 24 hours from a robots.txt update for its systems to adjust. Anthropic supports a Crawl-delay setting for slowing a bot down, and advises against blocking by IP address, because that can stop the bot from reading your robots.txt.

There is also a layer outside the file. On July 1, 2025, Cloudflare announced it was changing its default to block AI crawlers unless they pay creators, so check the setting rather than assume. If your site runs behind Cloudflare, or a host or plugin with a similar feature, bots may be turned away even when your robots.txt says nothing. For the underlying definitions, see AI crawler.

The training bot with the most attention is GPTBot.

The search bot with the most at stake for patient discovery is OAI-SearchBot.

There are also two things robots.txt cannot do. It cannot hide a page from patients, and it is not a way to keep a page out of Google, which Google says calls for noindex or password protection. If something is truly confidential, such as a patient portal, it belongs behind a login and not on a public page with a polite request attached.

Example: AI bot blocking at a health practice

Consider a naturopathic clinic with two locations. After reading a news story, the owner asks the developer to "block all the AI bots." This example is illustrative, not a transcript of a real site.

The developer sends back a file with five AI bot rules. Here they are.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The owner feels covered. The clinic's pages still rank on Google, because Googlebot has no rule. But two of those five rules are search bots. Under OpenAI's description, the clinic will not be shown in ChatGPT search answers. Under Anthropic's description, blocking Claude-SearchBot may reduce its visibility in user search results. A patient who asks either tool for a naturopath nearby is working with fewer chances to see the clinic.

The owner did not intend that. The goal was to keep clinical writing out of model training. That goal needed three rules, GPTBot, ClaudeBot and Google-Extended.

The fix takes about ten minutes. The developer deletes the two search bot rules, publishes the file, and sends the owner the live URL. The owner reads it, notes the date, and sets a quarterly reminder. After about 24 hours, the owner tests a few real patient questions in a ChatGPT temporary chat, running each several times because answers vary, and logs what appears.

A second clinic, with mostly public marketing pages, makes the opposite call. It leaves all of the bots allowed, on purpose, because it wants the widest possible exposure. That is a legitimate choice as well.

The point of both stories is the same. The file should reflect a decision that someone made and wrote down, and the decision should treat training and search as separate questions.

How to check yours in 60 seconds

1. Open yourpractice.com/robots.txt in a browser.
2. Find lines that say User-agent followed by a bot name.
3. Read the line beneath each. "Disallow: /" means that bot is told to stay out. "Allow: /" or nothing means it may read the site.
4. Look for "User-agent: *" followed by "Disallow: /". That tells every bot to stay out, including search engines.
5. Ask your developer whether Cloudflare or a security plugin blocks AI bots separately.
6. Write the date and what you found in your site notes.

What to do about it

1. Split the decision in two. Training bots are GPTBot, ClaudeBot and the Google-Extended token. Search bots are OAI-SearchBot and Claude-SearchBot.
2. For most practices that want new patients, keep the search bots allowed.
3. Block the training bots only if you choose to, with one rule per bot name.
4. Have your developer make the change, since some site platforms do not let you edit this file directly.
5. Expect a delay. OpenAI says its systems can take about 24 hours to adjust.
6. Test afterward with real patient questions in a ChatGPT temporary chat, running each question several times because answers vary.
7. Reread the live file every quarter and after any rebuild or plugin change.
8. Check any security service in front of your site, such as Cloudflare, for its own AI crawler setting, since it can override what the file says.
9. Keep a short note of who decided what and when, so a future redesign starts from your decisions and not from a blog post's block list.
10. Go one step past bots and list your content types: public service pages, downloadable forms and PDFs, the patient portal, online scheduling, any staging copy of the site, and subdomains. Mark each as open, limited or private, then compare that list to what the live file and your security settings actually do. Contradictions are common, such as a marketing team that wants visibility while a security plugin quietly turns bots away. Name one person who owns the file and approves changes, and put the whole policy on one page.
11. Check that a staging or test copy of the site is not open to search. A leftover copy can show outdated prices and providers, and indexing problems like that are easy to miss because nobody visits the copy.