Search + AI guide
OAI-SearchBot, GPTBot and ChatGPT-User: which to allow and which to block
- Author
- By Redenn Engineering
- Reviewed by
- Reviewed by Redenn Editorial
- Published
- Published
- Last reviewed
- Reviewed
- Reading time
- 5 min read
The short answer
OAI-SearchBot is the crawler OpenAI uses to find and link to pages in ChatGPT search results; GPTBot is the crawler that collects content that may be used to train models; ChatGPT-User fetches pages when a user asks ChatGPT to act on a link. They are controlled separately in robots.txt, so you can appear in ChatGPT search while opting out of training, and blocking the wrong one by accident removes you from a discovery channel.
The three OpenAI crawlers
These are the names and purposes OpenAI publishes on its bot documentation page as of 2026-09-24. Names and behaviour can change, so check the page before editing robots.txt.
| Crawler | What OpenAI says it is for | Effect of blocking it |
|---|---|---|
| OAI-SearchBot | ChatGPT search results and linking to sites | Your pages stop being surfaced and linked in ChatGPT search answers |
| GPTBot | Crawling content that may be used for training models | Your content is not collected for training; no effect on ChatGPT search visibility |
| ChatGPT-User | User-initiated actions, such as a user asking ChatGPT to open or summarize a specific link | A user asking ChatGPT about your page may be unable to fetch it |
robots.txt examples for each goal
In the table, each directive separated by a semicolon goes on its own line in the file, and each User-agent group is separated by a blank line. Rules are matched per crawler, so a crawler not named in any group follows the default group, if there is one.
| Goal | robots.txt rules | Notes |
|---|---|---|
| Appear in ChatGPT search, opt out of training | User-agent: OAI-SearchBot; Allow: / and User-agent: GPTBot; Disallow: / | The most common choice for a business website |
| Allow both search and training | User-agent: OAI-SearchBot; Allow: / and User-agent: GPTBot; Allow: / | Or leave them unnamed; an unlisted crawler follows the default group |
| Block both | User-agent: OAI-SearchBot; Disallow: / and User-agent: GPTBot; Disallow: / | You disappear from ChatGPT search; only sensible for private or paid content |
| Keep a private area out of everything | User-agent: *; Disallow: /portal/ | Applies to every well-behaved crawler, including search engines |
| Allow search but keep one section out of training | User-agent: GPTBot; Disallow: /guides/ | Path rules work per crawler |
Do not block OAI-SearchBot by accident
Most accidental blocks are not written on purpose. They come from a security plugin or firewall rule that blocks 'AI bots' as a category, a CDN bot-management setting that challenges unknown user agents, a robots.txt template copied from a site that chose to block everything, or a wildcard Disallow left over from staging. Each of these can hide a business from ChatGPT search without anyone noticing, because there is no error, only silence.
- Open yourdomain/robots.txt in a browser and read every group, not only the first one.
- Check firewall and CDN bot rules for category blocks that include OAI-SearchBot.
- Check that staging rules (Disallow: / for everyone) did not ship to production.
- Confirm the pages you want cited are not marked noindex and are listed in the sitemap.
GPTBot is a separate decision
Whether to allow GPTBot is a policy decision about training, not a visibility decision. Allowing it means your public content may be used to train models; blocking it does not affect ChatGPT search. Reasonable businesses land on either side, and Redenn asks the question rather than deciding for you.
Pros
- Allowing GPTBot: your public business information may be better represented in future models
- Allowing GPTBot: one less rule to maintain
Cons
- Allowing GPTBot: your content may be used in training without attribution or payment
- Either way: paid, proprietary or client-confidential material should never be crawlable by anyone, and belongs behind authentication rather than behind robots.txt
Verify crawler traffic by IP range, not user-agent alone
Anyone can send a request with the user-agent string 'OAI-SearchBot'. Scrapers do exactly that to slip past rules that let known bots through. OpenAI publishes the IP address ranges its crawlers use alongside its bot documentation; a request claiming to be OAI-SearchBot from an address outside those ranges is not OAI-SearchBot. The same principle applies to Googlebot and Bingbot, which publish their own verification methods.
- In server or CDN logs, group requests by user-agent, then check the source IPs against the published ranges.
- Treat unverified 'bot' traffic as ordinary traffic, subject to the same rate limits as everyone else.
- Do not build allow rules on user-agent strings alone; pair them with IP verification where your platform supports it.
What Redenn sets by default
In the Search + AI Foundation (CA$299) Redenn writes an explicit crawler policy: OAI-SearchBot and the major search crawlers are allowed, private paths are disallowed for everyone, and the GPTBot line is set to whatever you decide after a short conversation about training. The policy is kept in the site's code repository so it survives a redesign, and it is reviewed when OpenAI or other vendors change their crawler names.
Limitations
robots.txt is a request, not enforcement. Well-behaved crawlers honour it; others may not, which is why confidential content belongs behind a login. Allowing OAI-SearchBot does not make ChatGPT cite you; the page still has to be the best available answer. And crawler names change: the table above is accurate as of the access date and should be re-checked against OpenAI's documentation before you rely on it.
Questions
Should I block GPTBot?
That is a decision about whether your public content may be used for training, and it has no effect on ChatGPT search visibility. Decide it on its own merits and set the rule explicitly either way.
Will blocking GPTBot remove me from ChatGPT search?
No. ChatGPT search uses OAI-SearchBot. Block GPTBot and allow OAI-SearchBot if you want search visibility without training.
What is ChatGPT-User?
OpenAI's name for fetches triggered by a user action, such as asking ChatGPT to open or summarize a specific link. It is documented and controlled separately from the two crawlers.
How do I know whether OAI-SearchBot visits my site?
Look for its user-agent in server or CDN logs, then confirm the source IP is inside the ranges OpenAI publishes. A matching user-agent from an unpublished IP is not the real crawler.
Does robots.txt stop scrapers?
No. It is honoured by well-behaved crawlers only. Anything you must keep private belongs behind authentication, not behind a Disallow line.
Do I need to block AI bots to protect my content?
Public marketing content is meant to be read and cited; blocking search crawlers only makes you invisible. Reserve blocking for the training crawler if that is your policy, and for private areas for every crawler.
Sources
- Overview of OpenAI crawlers, OpenAI. Accessed 2026-09-24.
Related
Services
- Search + AI VisibilityMonthly plan covering technical SEO, citable content, entity consistency and AI-crawler readiness, from CA$699.
- Technical SEOMetadata, canonicals, sitemap, robots policy, structured data and Core Web Vitals, CA$99 on a Redenn build.
- Website SecurityStatic builds, SSL, updates, access control and monitoring included with hosting; audits of other sites quoted.
- Technical SupportEmail support on every hosting plan, priority on Pro Managed, with fixes covered and changes quoted first.
Ready when you are.
Configure a website in minutes, or ask for a written quote. Quotes arrive within two business days.