Didn't make the #ChromeDevSummit this year? Catch all the content (and more!) in the Chrome Dev Summit 2019 playlist on our Chrome Developers YouTube Channel.

robots.txt is not valid

Overview

The robots.txt file tells search engines what pages of your site they can crawl. An invalid robots.txt configuration can cause 2 general types of problems:

  • Not crawling public pages, causing your relevant content to show up less in search results.
  • Crawling private pages, exposing private information in search results.

Recommendations

Expand the robots.txt is not valid audit in your report to learn why your robots.txt file is invalid.

Viewing the 'robots.txt is not valid' audit.
Figure 1. Viewing the robots.txt is not valid audit

Here is an explanation of common errors:

  • No user-agent specified. Put a User-agent directive before your Allow or Disallow directive.
  • Pattern should either be empty, start with "/" or "*". Start your Allow or Disallow directive with one of these characters, or leave it empty.
  • Unknown directive. The directive name listed in the Content column is not part of the robots.txt specification.
  • Invalid sitemap URL. The sitemap URL should begin with http, https, or ftp.
  • "$" should only be used at the end of the pattern. Note that the original robots.txt specification does not define the $ pattern, so it may not be recognized by all search engines. It is recognized by Google Search, however. See URL matching based on path values.

More information

Lighthouse does not ensure that your robots.txt file is in the correct location.

Sources:

Feedback

Was this page helpful?
Yes
What was the best thing about this page?
It helped me complete my goal(s)
Thank you for the feedback. If you have specific ideas on how to improve this page, please create an issue.
It had the information I needed
Thank you for the feedback. If you have specific ideas on how to improve this page, please create an issue.
It had accurate information
Thank you for the feedback. If you have specific ideas on how to improve this page, please create an issue.
It was easy to read
Thank you for the feedback. If you have specific ideas on how to improve this page, please create an issue.
Something else
Thank you for the feedback. If you have specific ideas on how to improve this page, please create an issue.
No
What was the worst thing about this page?
It didn't help me complete my goal(s)
Thank you for the feedback. If you have specific ideas on how to improve this page, please create an issue.
It was missing information I needed
Thank you for the feedback. If you have specific ideas on how to improve this page, please create an issue.
It had inaccurate information
Thank you for the feedback. If you have specific ideas on how to improve this page, please create an issue.
It was hard to read
Thank you for the feedback. If you have specific ideas on how to improve this page, please create an issue.
Something else
Thank you for the feedback. If you have specific ideas on how to improve this page, please create an issue.