# Formalizing the Robots Exclusion Protocol Specification

Monday, July 01, 2019


For 25 years, the Robots Exclusion Protocol (REP) has been one of the most basic and critical
components of the web. It allows website owners to exclude automated clients, for example
[web crawlers](https://en.wikipedia.org/wiki/Web_crawler), from
accessing their sites - either partially or completely.


In 1994, [Martijn Koster](https://en.wikipedia.org/wiki/Martijn_Koster)
(a webmaster himself) created the initial standard after crawlers were overwhelming his site. With
more input from other webmasters, the REP was born, and it was adopted by search engines to help
website owners manage their server resources easier.


However, the REP was never turned into an official
[Internet standard](https://www.ietf.org/standards/), which means
that developers have interpreted the protocol somewhat differently over the years. And since its
inception, the REP hasn't been updated to cover today's corner cases. This is a challenging
problem for website owners because the ambiguous de-facto standard made it difficult to write
the rules correctly.


We wanted to help website owners and developers create amazing experiences on the internet instead
of worrying about how to control crawlers. Together with the original author of the protocol,
webmasters, and other search engines, we've documented how the REP is used on the modern web, and
submitted it to the IETF.


The proposed REP draft reflects over 20 years of real world experience of relying on robots.txt
rules, used both by Googlebot and other major crawlers, as well as about half a billion websites
that rely on REP. These fine grained controls give the publisher the power to decide what they'd
like to be crawled on their site and potentially shown to interested users. It doesn't change
the rules created in 1994, but rather defines essentially all undefined scenarios for robots.txt
parsing and matching, and extends it for the modern web. Notably:

1. Any [URI based transfer protocol](https://en.wikipedia.org/wiki/List_of_URI_schemes) can use robots.txt. For example, it's not limited to HTTP anymore and can be used for FTP or [CoAP](https://en.wikipedia.org/wiki/Constrained_Application_Protocol) as well.
2. Developers must parse at least the first 500 [kibibytes](https://en.wikipedia.org/wiki/Kibibyte) of a robots.txt. Defining a maximum file size ensures that connections are not open for too long, alleviating unnecessary strain on servers.
3. A new maximum caching time of 24 hours or cache directive value if available, gives website owners the flexibility to update their robots.txt whenever they want, and crawlers aren't overloading websites with robots.txt requests. For example, in the case of HTTP, Cache-Control headers could be used for determining caching time.
4. The specification now provisions that when a previously accessible robots.txt file becomes inaccessible due to server failures, known disallowed pages are not crawled for a reasonably long period of time.


Additionally, we've updated the
[augmented Backus-Naur form](https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_form)
in the internet draft to better define the syntax of robots.txt, which is critical for developers
to parse the lines.


RFC stands for Request for Comments, and we mean it: we uploaded the draft to IETF to get
feedback from developers who care about the basic building blocks of the internet. As we work to
give web creators the controls they need to tell us how much information they want to make
available to Googlebot, and by extension, eligible to appear in Search, we have to make sure we
get this right.


If you'd like to drop us a comment, ask us questions, or just say hi, you can find us on
[Twitter](https://twitter.com/googlesearchc) and in our
[Webmaster Community](https://support.google.com/webmasters/community),
both [offline](https://developers.google.com/search/events) and online.


Posted by [Henner Zeller](https://googlebot.com/),
[Lizzi Sassman](https://www.okaylizzi.com/+), and
[Gary Illyes](https://garyillyes.com/+)