Googlebot and the 15 MB thing

Tuesday, June 28, 2022

Over the last few days we've received a great deal of questions about a recent update to our documentation about Googlebot. Namely, we've documented that Googlebot only ever "sees" the first 15 megabytes (MB) when fetching certain file types. This threshold is not new; it's been around for many years. We just added it to our documentation because it might be helpful for some folks when debugging, and because it rarely ever changes.

This limit only applies to the bytes (content) received for the initial request Googlebot makes, not the referenced resources within the page.

For example, when you open, your browser will initially download the bytes of the HTML file, and based on those bytes it might make further requests for external JavaScript, images, or whatever else is referenced with a URL in the HTML. Googlebot does the same thing.

What does t