# Webmaster help center updates

June 13, 2006


We recently updated our
[webmaster help center](https://www.google.com/support/webmasters).
Two new sections that you may find particularly useful are:

- Using a robots.txt file
- Understanding HTTP status codes

## Using a robots.txt file


We've added a new section of help topics in the
[How Google crawls my site](https://developers.google.com/search/docs/fundamentals/how-search-works#crawling)
section. [These topics](https://developers.google.com/search/docs/crawling-indexing/robots/intro) include
information on:

- [How to create a robots.txt file](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt)
- [Descriptions of each user-agent that Google uses](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)
- [How to use pattern matching](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt#url-matching-based-on-path-values)
- [How often we recrawl your robots.txt file](https://developers.google.com/search/docs/crawling-indexing/robots/submit-updated-robots-txt#refresh-googles-robots.txt-cache) (around once a day)

## Understanding HTTP status codes


This section
[explains HTTP status codes](https://developers.google.com/search/docs/crawling-indexing/http-network-errors)
that your server might return when we request a page of your site. We display HTTP status codes
in several places in Google Sitemaps (such as on the
[robots.txt analysis page](https://www.google.com/webmasters/tools/robots-testing-tool)
and on the
[crawl errors](https://support.google.com/webmasters/answer/9679690)
page) and some site owners have asked us to provide more information about what these mean.

Posted by [Vanessa Fox](https://www.vanessafox.com/)