robots.txt
A text file at a site's root that tells automated crawlers which parts of the site they may or may not access.
What is robots.txt?
robots.txt is a plain text file placed at the root of a website (like example.com/robots.txt) that tells automated bots which parts of the site they're allowed to visit and which parts they should stay out of. A site owner might use it to say "search engines can crawl everything except our internal admin pages." It relies on the bot's own good faith to follow the rules, it isn't a security barrier, but the vast majority of legitimate crawlers, including nearly all major AI companies' bots, respect it.
Why it matters
robots.txt implements the Robots Exclusion Protocol, an informal standard originally created by Martijn Koster in 1994 that functioned as a de facto web convention for over two decades before Google worked with Koster and others to formalize it through the IETF, resulting in RFC 9309, published in September 2022, which standardized the file's syntax, error handling, and caching behavior. The file uses "User-agent" lines to target specific bots by name and "Disallow"/"Allow" lines to specify paths, and in the AI era it has become the primary lever site owners use to control AI crawler access, with companies adding their own named product tokens (GPTBot, Google-Extended, ClaudeBot, PerplexityBot, and others) so a site can permit or block each AI company's crawling independently.