Training Crawler
A crawler that collects web content to train a future version of an AI model, rather than to answer a live question.
What is Training Crawler?
A training crawler is a bot that collects web content specifically to help train a future version of an AI model, rather than to answer any live question right now. The pages it fetches today might not influence anything an AI says until a new model version is built and released, often months later. OpenAI's GPTBot is a well-known example: content it gathers is fed into the pipeline used to build OpenAI's next foundation models.
Why it matters
Training crawlers are distinguished from retrieval crawlers in that their output feeds offline model training or fine-tuning pipelines, not live inference, which is why site owners can often opt out of training crawlers without any immediate effect on how they appear in that same company's live AI answers. OpenAI documents GPTBot explicitly as being used to improve future models and offers a separate robots.txt directive for it distinct from its live-search bot. Google's equivalent control, Google-Extended, works differently: it is a robots.txt token, not a separate fetching bot, applied to content already crawled by Googlebot to determine eligibility for future Gemini model training and grounding, and Google states that blocking it has no effect on standard Search ranking.