This page describes the traffic from Google Events content accuracy crawlers.
Context
Google Events quality team implements crawlers. Crawlers are scripts that automate a web browser to perform some clicks and extract event information for quality testing purposes.
The Google events crawlers verify the partner links. The crawlers check that when a user clicks on a partner link on Google Search they are redirected to a partner page that meets the following landing page requirements:
- The page is fully functional
- The page shows the same event as the one displayed on Google
- The selected event is bookable
- The price on the partner page matches the price shown on Google
- The venue location and performer information on the partner page match the information the partner provides to Google in their feed
Types of crawlers
Google operates different kinds of accuracy crawlers for different purposes. All share the same set of egress IP ranges and User-Agent.
Price accuracy crawlers
Price accuracy crawlers verify price and availability information against the feed data, performing clicks until reaching the page where the user would have to enter personal or payment details.
Landing page crawlers
Landing page crawlers verify whether a landing page is functional, and can optionally verify other information on the page such as event name, venue, and artist. The crawler will not perform any clicks on the landing page.
These crawlers will generally perform a higher number of queries per day than price accuracy crawlers to ensure that all URLs displayed on Google are functional and previously removed content due to page errors can be automatically reinstated when the page becomes functional.
HTTP error codes
A landing page URL is considered to be in an error state if the first HTTP response after following redirects returns a response code in the 4xx or 5xx range.
Subsequent HTTP requests triggered asynchronously when loading the page are not taken into account for error detection. It is therefore important for the first HTTP response not to have a status code in the ranges described above, if the page is actually functional.
Number of queries
Landing page crawlers have a daily query limit equal to the combined number of URLs that were clicked at least once within the past 24 hours and all previously errored URLs, distributed evenly throughout the day.
Price accuracy crawlers are limited to 5,000 queries per day, distributed evenly throughout the day (approximately one query every 17 seconds). These crawlers simulate user behavior by:
Clicking the partner link displayed on Google
Selecting the desired event (if not already pre-selected)
Clicking 'continue' until they reach the page requesting personal/payment information
Extracting the final price details from the page
Range of IPs
The following IPv4 ranges are the complete list of IPs which may be used by our Google web crawlers.
34.116.24.0/23
34.124.108.0/22
64.233.172.0/23
66.102.6.0/23
66.102.8.0/23
66.249.80.0/22
66.249.84.0/23
66.249.88.0/24
66.249.93.0/24
74.125.208.0/21
142.250.32.0/23
192.178.8.0/21
These IP addresses are mapped to the following forward / reverse DNS addresses:
A.B.C.D => google-proxy-A-B-C-D.google.com
See also Google Webmaster documentation on how to identify traffic from Google Bot using reverse DNS.
User agent
The exact user agent used by the crawler as of 2024-03-10 is:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google-Events) Chrome/122.0.6261.94 Safari/537.36
HTTP caching
For purposes of reducing load on the partner website, our crawlers are generally configured to respect all standard http caching headers present in the response. That means that for correctly configured websites Google avoids repeatedly fetching content that changes rarely (e.g. JavaScript libraries).
A note on currency
When extracting prices, our crawlers need to unambiguously identify in what currency the price is expressed.
Currency symbols are ambiguous. For example, the dollar sign is used for more than 20 currencies (see Wikipedia pages for Dollar or Peso).
To be able to unambiguously identify the currency, our crawlers need to extract the 3 letters ISO-4217 currency code.
Relying on strings like "Price in Canadian dollars" makes it hard for our crawlers when the partner website is available in different languages, and is very brittle to changes in the website.
The preferred solution is to have the 3 letters code visible to the user, for example:
<div id="totalPrice">
123.05$ (CAD)
<div>
An alternative is to have the currency code available in the DOM, for example:
<div id="totalPrice" data-currency-code="CAD">
123.05$
<div>