Stay organized with collections
Save and categorize content based on your preferences.
Tuesday, May 27, 2014
The
Fetch as Google feature in Webmaster Tools
provides webmasters with the results of Googlebot attempting to fetch their pages. The server
headers and HTML shown are useful to diagnose technical problems and hacking side-effects, but
sometimes make double-checking the response hard: Help! What do all of these codes mean? Is
this really the same page as I see it in my browser? Where shall we have lunch? We can't help
with that last one, but for the rest, we've recently expanded this tool to also show how Googlebot
would be able to render the page.
Viewing the rendered page
In order to render the page, Googlebot will try to find all the external files involved, and fetch
them as well. Those files frequently include images, CSS and JavaScript files, as well as other
files that might be indirectly embedded through the CSS or JavaScript. These are then used to
render a preview image that shows Googlebot's view of the page.
You can find the
Fetch as Google feature
in the Crawl section of
Google Webmaster Tools.
After submitting a URL with "Fetch and render," wait for it to be processed (this might take a
moment for some pages). Once it's ready, just click on the response row to see the results.
Handling resources blocked by robots.txt
Googlebot follows the robots.txt rules
for all files that it fetches. If you are disallowing crawling of some of these files (or if they
are embedded from a third-party server that's disallowing Googlebot's crawling of them), we won't
be able to show them to you in the rendered view. Similarly, if the server fails to respond or
returns errors, then we won't be able to use those either (you can find similar issues in the
Crawl Errors
section of Webmaster Tools). If we run across either of these issues, we'll show them below the
preview image.
We recommend making sure Googlebot can access any embedded resource that meaningfully contributes
to your site's visible content, or to its layout. That will make Fetch as Google easier for you
to use, and will make it possible for Googlebot to find and index that content as well. Some types
of content—such as social media buttons, fonts or website-analytics scripts—tend not
to meaningfully contribute to the visible content or layout, and can be left disallowed from
crawling. For more information, please see our previous blog post on
how Google is working to understand the web better.
We hope this update makes it easier for you to diagnose these kinds of issues, and to discover
content that's accidentally blocked from crawling. If you have any comments or questions, let us
know here or drop by in the
webmaster help forum.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],[],[[["\u003cp\u003eGoogle Webmaster Tools now includes a "Fetch and Render" feature, allowing webmasters to see how Googlebot renders their pages, including handling of external resources like images, CSS, and JavaScript.\u003c/p\u003e\n"],["\u003cp\u003eThe rendered view helps diagnose technical issues and understand how Googlebot perceives a page's content and layout, especially concerning blocked resources.\u003c/p\u003e\n"],["\u003cp\u003eGoogle recommends allowing Googlebot access to crucial embedded resources that impact content visibility and layout for better rendering and indexing.\u003c/p\u003e\n"],["\u003cp\u003eResources like social media buttons or analytics scripts, which don't significantly impact visible content, can remain disallowed from crawling.\u003c/p\u003e\n"],["\u003cp\u003eThis update aims to simplify issue diagnosis and identify content unintentionally blocked from Google's crawling process.\u003c/p\u003e\n"]]],["The Fetch as Google feature, found in Google Webmaster Tools, allows users to see how Googlebot renders a webpage. This tool fetches external files like images, CSS, and JavaScript to create a preview image. Users can submit a URL using \"Fetch and render\" and view the results after processing. It follows robots.txt rules, and resources blocked or with server errors won't be rendered. Ensuring Googlebot can access essential resources improves the tool's effectiveness and content indexing.\n"],null,["# Rendering pages with Fetch as Google\n\n| It's been a while since we published this blog post. Some of the information may be outdated (for example, some images may be missing, and some links may not work anymore). Check how Google renders your page with the [URL inspection tool](https://search.google.com/search-console#view_screenshot) or [Rich Results Test](https://search.google.com/test/rich-results).\n\nTuesday, May 27, 2014\n\n\nThe\n[Fetch as Google feature in Webmaster Tools](https://support.google.com/webmasters/answer/158587)\nprovides webmasters with the results of Googlebot attempting to fetch their pages. The server\nheaders and HTML shown are useful to diagnose technical problems and hacking side-effects, but\nsometimes make double-checking the response hard: *Help! What do all of these codes mean? Is\nthis really the same page as I see it in my browser? Where shall we have lunch?* We can't help\nwith that last one, but for the rest, we've recently expanded this tool to also show how Googlebot\nwould be able to render the page.\n\nViewing the rendered page\n-------------------------\n\n\nIn order to render the page, Googlebot will try to find all the external files involved, and fetch\nthem as well. Those files frequently include images, CSS and JavaScript files, as well as other\nfiles that might be indirectly embedded through the CSS or JavaScript. These are then used to\nrender a preview image that shows Googlebot's view of the page.\n\n\nYou can find the\n[Fetch as Google feature](https://www.google.com/webmasters/tools/googlebot-fetch)\nin the Crawl section of\n[Google Webmaster Tools](https://search.google.com/search-console).\nAfter submitting a URL with \"Fetch and render,\" wait for it to be processed (this might take a\nmoment for some pages). Once it's ready, just click on the response row to see the results.\n\nHandling resources blocked by robots.txt\n----------------------------------------\n\n\nGooglebot follows the [robots.txt rules](/search/docs/crawling-indexing/robots/intro)\nfor all files that it fetches. If you are disallowing crawling of some of these files (or if they\nare embedded from a third-party server that's disallowing Googlebot's crawling of them), we won't\nbe able to show them to you in the rendered view. Similarly, if the server fails to respond or\nreturns errors, then we won't be able to use those either (you can find similar issues in the\n[Crawl Errors](https://support.google.com/webmasters/answer/35120)\nsection of Webmaster Tools). If we run across either of these issues, we'll show them below the\npreview image.\n\n\nWe recommend making sure Googlebot can access any embedded resource that meaningfully contributes\nto your site's visible content, or to its layout. That will make Fetch as Google easier for you\nto use, and will make it possible for Googlebot to find and index that content as well. Some types\nof content---such as social media buttons, fonts or website-analytics scripts---tend not\nto meaningfully contribute to the visible content or layout, and can be left disallowed from\ncrawling. For more information, please see our previous blog post on\n[how Google is working to understand the web better](/search/blog/2014/05/understanding-web-pages-better).\n\n\nWe hope this update makes it easier for you to diagnose these kinds of issues, and to discover\ncontent that's accidentally blocked from crawling. If you have any comments or questions, let us\nknow here or drop by in the\n[webmaster help forum](https://support.google.com/webmasters/threads?hl=en&thread_filter=(category:search_console)).\n\nPosted by Shimi Salant, Webmaster Tools team"]]
