# Events accuracy crawlers

This page describes the traffic from Google Events content accuracy crawlers.

## Context

Google Events quality team implements crawlers. Crawlers are scripts that
automate a web browser to perform some clicks and extract event information for
**quality testing purposes**.

The Google events crawlers verify the partner links. The crawlers check that
when a user clicks on a partner link on Google Search they are redirected to a
partner page that meets the following
[landing page requirements](https://developers.google.com/actions-center/verticals/events/policies/integration_policies#landing_page_requirements):

- The page is fully functional
- The page shows the same event as the one displayed on Google
- The selected event is bookable
- The price on the partner page matches the price shown on Google
- The venue location and performer information on the partner page match the information the partner provides to Google in their feed

## Types of crawlers

Google operates different kinds of accuracy crawlers for different purposes. All
share the same set of egress IP ranges and User-Agent.

### Price accuracy crawlers

Price accuracy crawlers verify price and availability information against the
feed data, performing clicks until reaching the page where the user would have
to enter personal or payment details.

### shallow link (404) crawlers

Landing page crawlers verify whether a landing page is functional, and can
optionally verify other information on the page such as event name, venue, and
artist. The crawler won't perform any clicks on the landing page.

These crawlers will generally perform a higher number of queries per day than
price accuracy crawlers to ensure that all URLs displayed on Google are
functional and previously removed content due to page errors can be
automatically reinstated when the page becomes functional.

## HTTP error codes

A landing page URL is considered to be in an error state if the first HTTP
response after following redirects returns a response code in the 4xx or 5xx
range.

Subsequent HTTP requests triggered asynchronously when loading the page are not
taken into account for error detection. It is therefore important for the first
HTTP response not to have a status code in the ranges described above, if the
page is actually functional.

## Number of queries

Landing page crawlers have a daily query limit equal to the combined number of
URLs that were clicked at least once within the past 24 hours and all previously
errored URLs, distributed evenly throughout the day.

Price accuracy crawlers are limited to 5,000 queries per day, distributed evenly
throughout the day (approximately one query every 17 seconds). These crawlers
simulate user behavior by:

- Clicking the partner link displayed on Google

- Selecting the chosen event (if not already pre-selected)

- Clicking 'continue' until they reach the page requesting personal or payment
  information

- Extracting the final price details from the page

## Range of IPs

The following IPv4 ranges are the complete list of IPs which may be used by our
Google web crawlers.

    34.116.24.0/23
    34.124.108.0/22
    64.233.172.0/23
    66.102.6.0/23
    66.102.8.0/23
    66.249.80.0/22
    66.249.84.0/23
    66.249.88.0/24
    66.249.93.0/24
    74.125.208.0/21
    142.250.32.0/23
    146.148.114.0/23
    172.217.80.0/20
    192.179.0.0/20
    192.178.8.0/21

These IP addresses are mapped to the following forward / reverse DNS addresses:

A.B.C.D =\> google-proxy-A-B-C-D.google.com

See also [Google Webmaster documentation on how to identify traffic from Google
Bot using reverse
DNS](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot).

## User agent

The exact user agent used by the crawler as of 2024-03-10 is:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko;
Google-Events) Chrome/122.0.6261.94 Safari/537.36

> [!IMPORTANT]
> **Important:** The user agent changes every time the crawler is updated to a newer Chrome version, but the string **Google-Events** is always present.

## HTTP caching

For purposes of reducing load on the partner website, our crawlers are generally
configured to respect all standard http caching headers present in the response.
That means that for correctly configured websites Google avoids repeatedly
fetching content that changes rarely (e.g. JavaScript libraries).

## A note on currency

When extracting prices, our crawlers need to unambiguously identify in what
currency the price is expressed.

Currency symbols are ambiguous. For example, the
[dollar sign](https://en.wikipedia.org/wiki/Dollar_sign) is used for more than
20 currencies (see Wikipedia pages for
[Dollar](https://en.wikipedia.org/wiki/Dollar) or
[Peso](https://en.wikipedia.org/wiki/Peso)).

To be able to unambiguously identify the currency, our crawlers need to extract
the 3 letters [ISO-4217 currency code](https://en.wikipedia.org/wiki/ISO_4217).

Relying on strings like "Price in Canadian dollars" makes it hard for our
crawlers when the partner website is available in different languages, and is
very brittle to changes in the website.

The preferred solution is to have the 3
letters code visible to the user, for example:

    <div id="totalPrice">
      123.05$ (CAD)
    <div>

An alternative is to have the currency code
available in the DOM, for example:

    <div id="totalPrice" data-currency-code="CAD">
      123.05$
    <div>