robots.txt file tells search engines what pages of your site they can crawl. An
robots.txt configuration can cause 2 general types of problems:
- Not crawling public pages, causing your relevant content to show up less in search results.
- Crawling private pages, exposing private information in search results.
Expand the robots.txt is not valid audit in your report to learn why your
robots.txt file is
Here is an explanation of common errors:
No user-agent specified. Put a
User-agentdirective before your
Pattern should either be empty, start with "/" or "*". Start your
Disallowdirective with one of these characters, or leave it empty.
Unknown directive. The directive name listed in the Content column is not part of the
Invalid sitemap URL. The sitemap URL should begin with
"$" should only be used at the end of the pattern. Note that the original
robots.txtspecification does not define the
$pattern, so it may not be recognized by all search engines. It is recognized by Google Search, however. See URL matching based on path values.
Lighthouse does not ensure that your
robots.txt file is in the correct location.
Was this page helpful?