Retrieval Crawler
A crawler that fetches web content in real time, specifically to answer a live user query.
What is Retrieval Crawler?
A retrieval crawler fetches web content in real time, specifically to help answer a live question someone just asked an AI system. Unlike a training crawler, which gathers data for some future model, a retrieval crawler's fetch is tied to a specific query happening right now, similar to how a search engine fetches fresh results the moment you search. OpenAI's OAI-SearchBot, which powers ChatGPT's live web search and citations, is a standard example.
Why it matters
Retrieval crawlers typically build and refresh a searchable index ahead of time, similar to a traditional search engine, so an AI system can quickly pull and cite current information the moment a user's query needs it, rather than fetching a page from scratch on every single query. OpenAI operates OAI-SearchBot for this purpose, kept deliberately separate in its documentation and robots.txt controls from GPTBot (training) and ChatGPT-User (one-off, user-triggered fetches), so a site can, for instance, allow itself to be cited in live ChatGPT search results while still opting out of AI training. This category has grown alongside the broader shift toward AI-generated answers with citations, seen in features like Google's AI Overviews and Perplexity's answer engine, both of which depend on similarly purpose-built retrieval bots.