What Is GPTBot?

GPTBot is OpenAI's web crawler that collects content which may be used to train its generative AI foundation models. For a health practice, it is the bot your robots.txt file speaks to when you decide whether your site content can feed future model training. OpenAI states that disallowing GPTBot indicates a site's content should not be used for that training.

How GPTBot works

A crawler is software that visits web pages and copies what it finds. GPTBot is one of three crawlers OpenAI documents on its bots page, and each has a separate job.

• GPTBot crawls content that may be used for training. OpenAI says it is used to make its generative AI foundation models more useful and safe.
• OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. This is the bot behind being shown in ChatGPT search. See OAI-SearchBot.
• ChatGPT-User handles actions a person starts in ChatGPT, such as opening a page. OpenAI says robots.txt rules may not apply to it, because a user initiated the visit.

OpenAI states that each of these settings is independent. If a site allows both GPTBot and OAI-SearchBot, OpenAI says it may use the results from just one crawl for both purposes, to avoid duplicate crawling.

Your control is the robots.txt file, a plain-text file at yourpractice.com/robots.txt. A rule names a bot and says whether it may read the site. Anyone can open the file, including your competitors, so write rules you are comfortable having read.

OpenAI's page also lists a user agent string for each bot, the long label the bot sends with each visit. Your host or developer can search server logs for the name GPTBot to see whether it has actually visited. Seeing a visit confirms the bot reached you. It does not tell you what was done with the content. OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust. Google's documentation notes that these files are advisory and cannot enforce crawler behavior. OpenAI publishes how its own bots are meant to act, so the rule is a clear signal, but it is not a lock.

Here is what a GPTBot opt-out looks like.

User-agent: GPTBot
Disallow: /

Those two lines affect GPTBot only. They do not touch OAI-SearchBot, Googlebot or any other bot. The broader topic is covered in AI crawler.

The full trade-off between training and search is in AI bot blocking. If you want a quick way to remember it, GPTBot is about what AI companies learn from your pages (the background knowledge a model learns), and OAI-SearchBot is about whether those companies can show your pages to a patient today.

One context point. On July 1, 2025, Cloudflare announced it was changing its default to block AI crawlers unless they pay creators. If your site sits behind Cloudflare, it may already be refusing some bots without a line in your robots.txt. Ask your developer which layer is doing the work.

Example: GPTBot at a health practice

Take a hormone clinic with a blog full of original clinical explainers written by its physician. The owner reads a news story about AI companies copying website text and asks the developer to "block OpenAI." This example is illustrative, not a transcript of a real site.

The developer reads OpenAI's bots page and sees GPTBot, OAI-SearchBot and ChatGPT-User. The developer asks a clarifying question. Which goal does the owner want?

The owner thinks it through. Patients should be able to find the clinic when they ask ChatGPT for hormone providers in the city. The physician's articles, though, should not feed future model training. That is two goals that map to two bots.

So the developer adds the two-line GPTBot rule above and leaves OAI-SearchBot alone. The owner opens the live robots.txt, sees the GPTBot block and no rule for OAI-SearchBot, and files a screenshot with the date.

Compare the version that goes wrong. A second developer, trying to be thorough, also blocks OAI-SearchBot. Now OpenAI's own page applies. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they may still appear as navigational links. The clinic has traded away a patient-finding path for a training preference it already had covered.

The physician also asks one more question, whether the opt-out affects her Google rankings. The sources the clinic read say GPTBot is an OpenAI control for training, and Google's crawlers are governed by their own rules, so the clinic keeps its Googlebot rules untouched and checks Search Console after the change.

A third clinic decides its public pages are marketing anyway. It sets no GPTBot rule and uses its effort elsewhere. That is also reasonable. The point of the example is not which choice wins. It is that each clinic's file matches what its owner decided.

How to check yours in 60 seconds

1. Open yourpractice.com/robots.txt in a browser.
2. Search the page for the word GPTBot.
3. If you see "User-agent: GPTBot" followed by "Disallow: /", you have asked it to stay out.
4. If GPTBot does not appear, no rule targets it, though a general "User-agent: *" rule may still apply to it.
5. Search the same page for OAI-SearchBot and confirm it is not blocked.
6. Note the date you checked and what you found, so you can compare after your next site update.

What to do about it

1. Decide whether you are comfortable with your public pages informing future model training. Write the answer in one sentence.
2. If you want to opt out, add a GPTBot-only rule using the two lines above.
3. Leave OAI-SearchBot allowed if you want ChatGPT search to be able to show you.
4. Ask your web developer to publish the change and send you the live file.
5. Allow about 24 hours for OpenAI's systems to adjust, then reread the file.
6. Check your host and any security service for a separate AI crawler setting.
7. Put a quarterly reminder in your calendar to reread the file, because rebuilds and plugins can rewrite it.
8. Keep a copy of the live file in your records, with the date.
9. Keep private material off the public web entirely. Robots.txt only asks well-behaved bots to stay away, so patient forms, portal pages and staging sites belong behind a login and not behind a robots rule.
10. If you opted out, tell your content team and your web vendor, so nobody undoes the rule during a redesign. If you stayed open, record that choice as well.