What Is an AI Crawler?

An AI crawler is an automated program that reads web pages on behalf of an AI company, either to train its models, to power its search features, or to fetch a page when a user asks. For a health practice, it matters because different crawlers do different jobs, and a rule meant for one can change whether AI tools can name you. Your website's robots.txt file is where you tell each one what it may read.

How AI crawlers work

A crawler, sometimes called a bot or spider, is software that follows links from page to page and copies what it finds. Search engines have used them for decades. AI companies now run their own, and most run more than one. The names look similar, which is part of the confusion.

A user agent is the label a bot sends with each request, a bit like a name tag. Your server logs record it, and robots.txt rules are matched against it.

Before a well-behaved crawler reads your site, it checks a plain-text file at yourpractice.com/robots.txt. The file lists bot names, called user agents, with a rule under each. "Disallow: /" under a bot's name means stay out of the whole site. Google's documentation describes the file as advisory. It says the instructions cannot enforce crawler behavior, and compliant crawlers like Googlebot simply honor them.

The key point is that each AI company splits its crawling into jobs.

• OpenAI lists GPTBot, which crawls content that may be used to train its models, and OAI-SearchBot, which surfaces sites in ChatGPT's search features. It also lists ChatGPT-User, used for actions a person starts in ChatGPT. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user triggers those visits.
• Anthropic lists ClaudeBot for training data, Claude-SearchBot for improving search result relevance, and Claude-User for fetching pages when someone asks Claude a question. Anthropic says the bots honor robots.txt and recommends against blocking by IP address, because that stops the bots from reading your robots.txt at all.
• Google uses a robots.txt token called Google-Extended. It is not a separate crawler. Google says it controls whether crawled content can be used for Gemini training and grounding, and that it does not affect inclusion in Google Search.

OpenAI states that each of its settings is independent of the others. That is the single most useful sentence for a practice owner, because it means you can allow one job and refuse another. The detail for each bot lives on its own page, starting with GPTBot for training.

The search side is covered under OAI-SearchBot.

The Google control is covered under Google-Extended.

Anthropic also gives site owners two extra tools. It supports the Crawl-delay setting, which asks a bot to slow down, and it publishes its crawler IP addresses. It advises against blocking by IP address, because that can stop the bots from reading your robots.txt at all. The simple path is a rule in robots.txt, one per bot name, for each subdomain you want to cover.

One more fact affects how practices land by default. On July 1, 2025, Cloudflare announced it was changing its default to block AI crawlers unless they pay creators for their content. If your site runs behind Cloudflare, check the setting, because a service you forgot about may be making this choice for you.

Example: AI crawlers at a health practice

Consider a chiropractic group with four locations. The owner reads that AI companies copy website text and asks the developer to "block the AI crawlers." This example is illustrative, not a transcript of a real site.

The developer, trying to be thorough, copies a block list from a blog post. It disallows GPTBot, ClaudeBot, Google-Extended and OAI-SearchBot. The site still ranks on Google, because Googlebot was left alone, so nothing looks wrong.

Weeks later, the group asks ChatGPT for a chiropractor for lower back pain in one of its cities and is not named. The cause is not certain, because many things decide who gets named. But one thing the group did is plain from OpenAI's own page. Sites that opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. The group removed itself from that path without realizing it.

Now compare a second owner who writes out the goal in one sentence first. "Appear in ChatGPT and Claude search answers. Opt out of model training." The developer turns that into two allowed search bots and two disallowed training bots. The owner opens robots.txt, reads it, and files a copy with the site records.

Neither owner changed what Google sees. The difference is that the second owner's rules match a decision a person actually made.

A third case is the owner who sets no rules at all. Every bot may read everything. For most cash-pay practices, whose service pages are public marketing anyway, that can be acceptable. It should still be a choice and not an accident.

How to check yours in 60 seconds

1. Type your domain followed by /robots.txt into a browser, such as yourpractice.com/robots.txt.
2. Look for lines that begin with User-agent and name GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot or Google-Extended.
3. Read the line under each. "Disallow: /" means that bot was told to stay out.
4. Also look for "User-agent: *" followed by "Disallow: /". That tells every bot to stay out, including search engines.
5. If the page is blank or missing, no special rules exist.
6. Write down what you found and the date, even if it is "no AI rules." That note is your baseline for the next check.

What to do about it

1. Write your goal in plain words, such as "appear in ChatGPT answers but stay out of AI model training."
2. Allow the search bots, OAI-SearchBot and Claude-SearchBot, if you want to be named by those tools.
3. Decide separately about the training bots, GPTBot and ClaudeBot. Also decide about Google-Extended.
4. Ask whoever manages your site to write one rule per bot name, and read the live file back using the check above.
5. If you use Cloudflare or a security plugin, open its AI crawler setting and confirm it matches your goal.
6. Allow about 24 hours for OpenAI's systems to adjust after you change the file. For the full trade-off between training and search, read AI bot blocking.
7. Recheck after any site rebuild, plugin update or host change, since these can rewrite the file.
8. Keep a one-page note of your decisions: which bots are allowed, which are blocked, who wrote the rules and when. It saves a lot of arguing the next time someone redesigns the site.
9. Keep private material out of reach in the right way. Robots.txt only guides crawlers that choose to follow it, so patient forms, portal pages and anything confidential belong behind a login. Also ask whether a test copy of your site or a PDF folder is publicly reachable, since those are the pages that tend to be forgotten.