Stay organized with collections
Save and categorize content based on your preferences.
February 06, 2006
Today, we released new features for Sitemaps.
robots.txt analysis
If the site has a
robots.txt file,
the new robots.txt tab
provides Googlebot's view
of that file, including when Googlebot last accessed it, the status it returns, and if it blocks
access to your home page. This tab also lists any syntax errors in the file.
You can enter a list of URLs to see if the robots.txt file allows or blocks them.
You can also test changes to your robots.txt file by entering them here and then testing them
against the Googlebot user-agent, other Google user agents, or the Robots Standard. This lets you
experiment with changes to see how they would impact the crawl of your site, as well as make sure
there are no errors in the file, before making changes to the file on your site.
If you don't have a robots.txt file, you can use this page to test a potential robots.txt file
before you add it to your site.
More stats
Crawl stats now include the page on your site that had the highest PageRank, by month, for the
last three months.
Page analysis now includes a list of the most common words in your site's content and in external
links to your site. This gives you additional information about why your site might come up for
particular search queries.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],[],[[["\u003cp\u003eGoogle released new Sitemaps features, including robots.txt analysis and enhanced crawl stats.\u003c/p\u003e\n"],["\u003cp\u003eThe robots.txt analysis tool allows you to view Googlebot's interpretation of your robots.txt file, test changes, and identify any errors.\u003c/p\u003e\n"],["\u003cp\u003eCrawl stats now display the page with the highest PageRank for the past three months, along with common words in your site's content and external links.\u003c/p\u003e\n"]]],["New Sitemaps features released include robots.txt analysis, offering Googlebot's view, access status, and syntax error detection. Users can test URLs against the file and experiment with changes before updating their site. Crawl stats now display the highest PageRank page per month for the past three months. Page analysis was enhanced to list frequent words within the site's content and external links, offering insights into search query relevance.\n"],null,["# More stats and analysis of robots.txt files\n\n| It's been a while since we published this blog post. Some of the information may be outdated (for example, some images may be missing, and some links may not work anymore).\n\nFebruary 06, 2006\n\nToday, we released new features for Sitemaps.\n\nrobots.txt analysis\n-------------------\n\n\nIf the site has a\n[robots.txt file](https://www.robotstxt.org/wc/exclusion.html),\nthe new robots.txt tab\n[provides Googlebot's view](https://support.google.com/webmasters/answer/6062598)\nof that file, including when Googlebot last accessed it, the status it returns, and if it blocks\naccess to your home page. This tab also lists any syntax errors in the file.\n\nYou can enter a list of URLs to see if the robots.txt file allows or blocks them.\n\n\nYou can also test changes to your robots.txt file by entering them here and then testing them\nagainst the Googlebot user-agent, other Google user agents, or the Robots Standard. This lets you\nexperiment with changes to see how they would impact the crawl of your site, as well as make sure\nthere are no errors in the file, before making changes to the file on your site.\n\n\nIf you don't have a robots.txt file, you can use this page to test a potential robots.txt file\nbefore you add it to your site.\n\nMore stats\n----------\n\n\nCrawl stats now include the page on your site that had the highest PageRank, by month, for the\nlast three months.\n\n\nPage analysis now includes a list of the most common words in your site's content and in external\nlinks to your site. This gives you additional information about why your site might come up for\nparticular search queries.\n\nPosted by [Vanessa Fox](https://www.vanessafox.com/)"]]
