Tuesday, June 28, 2022
Over the last few days we've received a great deal of questions about a recent update to our documentation about Googlebot. Namely, we've documented that Googlebot only ever "sees" the first 15 megabytes (MB) when fetching certain file types. This threshold is not new; it's been around for many years. We just added it to our documentation because it might be helpful for some folks when debugging, and because it rarely ever changes.
This limit only applies to the bytes (content) received for the initial request Googlebot makes, not the referenced resources within the page.
For example, when you open https://example.com/puppies.html, your browser will
initially download the bytes of the HTML file, and based on those bytes it might make further
requests for external JavaScript, images, or whatever else is referenced with a URL in the HTML.
Googlebot does the same thing.
What does t