OAI-SearchBot
OpenAI's crawler that fetches web content in real time to power live search-style answers inside ChatGPT.
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's crawler that fetches web content in real time so ChatGPT can give up-to-date, cited answers to live search-style questions. If a user asks ChatGPT about something happening this week, OAI-SearchBot is part of the machinery that finds and surfaces relevant, current web pages, similar to how a search engine's crawler builds the results you see after typing a query. It is a distinct bot from OpenAI's other crawlers, each serving a different purpose.
Why it matters
OpenAI documents three separate, independently controllable crawlers: GPTBot, which gathers content to train and improve OpenAI's foundation models; OAI-SearchBot, which crawls to build and refresh the index that powers AI-driven search results and citations inside ChatGPT; and ChatGPT-User, which fetches a specific page only when a live user explicitly asks ChatGPT to read or reference it, and is not used for automated crawling or training. A publisher can, for instance, disallow GPTBot to opt out of training data collection while still allowing OAI-SearchBot so their pages remain eligible to be surfaced and cited in ChatGPT's live search answers, since OpenAI's robots.txt directives for each bot operate independently of one another.