# Safe Browsing Update API (v4)

## Overview

The Update API lets your client applications download hashed versions of the Safe Browsing lists for
storage in a local database. URLs can then be checked locally. Only if a match is found in the
local database does the client need to send a request to the Safe Browsing servers to verify
whether the URL is included on the Safe Browsing lists.

Before using the Update API, you need to set up a local database. Safe Browsing provides a
Go package you can use to get going. For more details, see the Database Setup section under
[Local Databases](https://developers.google.com/safe-browsing/v4/local-databases#database-setup).

## Updating the local database

To stay current, clients are required to periodically update the Safe Browsing lists in their
local database. To save bandwidth, clients download the hash prefixes of URLs rather than the
raw URLs. For example, if "www.badurl.com/" is on a Safe Browsing list, clients download the
SHA256 hash prefix of that URL rather than the URL itself. In the majority of cases the hash
prefixes are 4 bytes long, meaning that the average bandwidth cost of downloading a single list
entry is 4 bytes before compression.

To update the Safe Browsing lists in the local database, send an HTTP `POST` request to the
[threatListUpdates.fetch](https://developers.google.com/safe-browsing/reference/rest/v4/threatListUpdates/fetch)
method:

- The HTTP `POST` request includes the names of the lists to be updated along with various client constraints to account for memory and bandwidth limitations.
- The HTTP `POST` response returns either a full update or a partial update. The response *may* also return a minimum wait duration.

## Example: threatListUpdates.fetch

### HTTP POST request

In the following example, the updates for a single Safe Browsing list are requested.

#### Request header

The request header includes the request URL and the content type. Remember to substitute your API
key for `API_KEY` in the URL.

```
POST https://safebrowsing.googleapis.com/v4/threatListUpdates:fetch?key=API_KEY HTTP/1.1
Content-Type: application/json
```

#### Request body


The request body includes the client information (ID and version) and the update information (the
list name, the list state, and client constraints).
For more details, see the
[threatListUpdates.fetch request body](https://developers.google.com/safe-browsing/reference/rest/v4/threatListUpdates/fetch#request-body)
and the explanations that follow the code example.

```
{
  "client": {
    "clientId":       "yourcompanyname",
    "clientVersion":  "1.5.2"
  },
  "listUpdateRequests": [{
    "threatType":      "MALWARE",
    "platformType":    "WINDOWS",
    "threatEntryType": "URL",
    "state":           "Gg4IBBADIgYQgBAiAQEoAQ==",
    "constraints": {
      "maxUpdateEntries":      2048,
      "maxDatabaseEntries":    4096,
      "region":                "US",
      "supportedCompressions": ["RAW"]
    }
  }]
}
```

###### Client information

The `clientID` and `clientVersion` fields should uniquely identify a client implementation, not an
individual user. (Client information is used in server-side logging. You can choose
any name for the client ID but we suggest you choose a name that represents the true identity of the
client, such as your company name, presented as all one word, in all-lowercase letters.)

###### Safe Browsing lists

The `threatType`, `platformType`, and `threatEntryType` fields
are combined to identify (name) the Safe Browsing lists. In the example, one list is identified:
MALWARE/WINDOWS/URL. Before sending a request, make sure the type combinations you specify are valid
(see [Safe Browsing Lists](https://developers.google.com/safe-browsing/v4/lists)).

###### Client state

The `state` field holds the current client state of the Safe Browsing list.
(Client states are returned in the `newClientState` field of the
[threatListUpdates.fetch response](https://developers.google.com/safe-browsing/v4/update-api#http-post-response).)
For initial updates, leave the `state` field empty.

###### Size constraints

The `maxUpdateEntries` field specifies the total number of updates that the client can manage (in
the example, 2048). The `maxDatabaseEntries` field specifies the total number of entries the local
database can manage (in the example, 4096). Clients should set size constraints
to protect memory and bandwidth limitations and to safeguard against list
growth. For more information,
(see [Update Constraints](https://developers.google.com/safe-browsing/v4/update-constraints)).

###### Supported compressions

The `supportedCompressions` field lists the compression types the client supports. In the
example, the client supports only raw, uncompressed data. Safe Browsing, however, supports additional
compression types (see [Compression](https://developers.google.com/safe-browsing/v4/compression)).

### HTTP POST response

In this example, the response returns a partial update for the Safe Browsing list using the
requested compression type.

#### Response header

The response header includes the [HTTP status code](https://developers.google.com/safe-browsing/v4/status-codes)
and the content type. Clients that receive a status code other than HTTP/200 must enter back-off mode
(see [Request Frequency](https://developers.google.com/safe-browsing/v4/request-frequency)).

```
HTTP/1.1 200 OK
Content-Type: application/json
```

#### Response body

The response body includes the update information (the list name, the response type, the
additions and removals to be applied to the local database, the new client
state, and a checksum). In the example, the response also includes a minimum wait duration. For more
details, see the
[threatListUpdates.fetch response body](https://developers.google.com/safe-browsing/reference/rest/v4/threatListUpdates/fetch#response-body)
and the explanations that follow the code example.

```
{
  "listUpdateResponses": [{
    "threatType":      "MALWARE",
    "threatEntryType": "URL",
    "platformType":    "WINDOWS",
    "responseType" :   "PARTIAL_UPDATE",
    "additions": [{
      "compressionType": "RAW",
      "rawHashes": {
        "prefixSize": 4,
        "rawHashes":  "rnGLoQ=="
      }
    }],
    "removals": [{
      "compressionType": "RAW",
      "rawIndices": {
        "indices": [0, 2, 4]
      }
    }],
    "newClientState": "ChAIBRADGAEiAzAwMSiAEDABEAFGpqhd",
    "checksum": {
      "sha256": "YSgoRtsRlgHDqDA3LAhM1gegEpEzs1TjzU33vqsR8iM="
    }
  }],
  "minimumWaitDuration": "593.440s"
}
```

###### Database updates

The `responseType` field will indicate a partial or full update. In the example, a partial update
is returned, so the response includes both additions and removals. There may be multiple sets of
additions, but only one set of removals
(see [Database Updates](https://developers.google.com/safe-browsing/v4/local-databases#database-updates)).

###### New client state

The `newClientState` field holds the new client state for the newly updated Safe Browsing list.
Clients must save the new client state for subsequent update requests (the `state` field in the
[threatListUpdates.fetch request](https://developers.google.com/safe-browsing/v4/update-api#http-post-request)
or the `clientStates` field in the
[fullHashes.find request](https://developers.google.com/safe-browsing/v4/update-api#http-post-request_1)).


###### Checksums

The checksum lets clients verify that the local database has not suffered any corruption. If the
checksum does not match, the client must clear the database and reissue an update with an empty
`state` field. However, clients in this situation must still follow the time intervals for updates
(see [Request Frequency](https://developers.google.com/safe-browsing/v4/request-frequency)).

###### Minimum wait durations

The `minimumWaitDuration` field indicates that the client must wait 593.44 seconds (9.89 minutes)
before sending another update request. Note that a wait period may or may not be included in the
response (see [Request Frequency](https://developers.google.com/safe-browsing/v4/request-frequency)).

## Checking URLs

To check if a URL is on a Safe Browsing list, the client must first compute the hash and hash
prefix of the URL (see [URLs and Hashing](https://developers.google.com/safe-browsing/v4/urls-hashing)). The client
then queries the local database to determine if there is a match. If the hash prefix **is
not present** in the local database, then the URL is considered safe (not on the Safe
Browsing lists).

If the hash prefix **is present** in the local database (a hash prefix collision),
the client must send the hash prefix to the Safe Browsing servers for verification.
The servers will return all full-length SHA 256 hashes that contain the given hash prefix.
If one of those full-length hashes matches the full-length hash of the URL in question,
then the URL is considered unsafe. If none of the full-length hashes match the full-length hash
of the URL in question, then that URL is considered safe.

At no point does Google learn about the URLs you are examining. Google does learn the hash
prefixes of URLs, but the hash prefixes don't provide much information about the actual URLs.

To check if a URL is on a Safe Browsing list, send an HTTP `POST` request to the
[fullHashes.find](https://developers.google.com/safe-browsing/reference/rest/v4/fullHashes/find)
method:

- The HTTP `POST` request can include up to 500 threat entries.
- The HTTP `POST` request includes the hash prefixes of the URLs to be checked. Clients are encouraged to batch multiple threat entries into a single request to lower bandwidth usage.
- The HTTP `POST` response returns the matching full-length hashes along with the positive and negative cache durations. The response *may* also return a minimum wait duration.

## Example: fullHashes.find

### HTTP POST request

In the following example, the names of two Safe Browsing lists and three hash prefixes are sent
for comparison and verification.

#### Request header

The request header includes the request URL and the content type. Remember to substitute
your API key for `API_KEY` in the URL.

```
POST https://safebrowsing.googleapis.com/v4/fullHashes:find?key=API_KEY HTTP/1.1
Content-Type: application/json
```

#### Request body


The request body includes the client information (ID and version), the client states, and
the threat information (the list names and the hash prefixes). For JSON requests, hash prefixes must
be sent in base64-encoded form.
For more details, see the
[fullHashes.find request body](https://developers.google.com/safe-browsing/reference/rest/v4/fullHashes/find#request-body)
and the explanations that follow the code example.

```
{
  "client": {
    "clientId":      "yourcompanyname",
    "clientVersion": "1.5.2"
  },
  "clientStates": [
    "ChAIARABGAEiAzAwMSiAEDABEAE=",
    "ChAIAhABGAEiAzAwMSiAEDABEOgH"
  ],
  "threatInfo": {
    "threatTypes":      ["MALWARE", "SOCIAL_ENGINEERING"],
    "platformTypes":    ["WINDOWS"],
    "threatEntryTypes": ["URL"],
    "threatEntries": [
      {"hash": "WwuJdQ=="},
      {"hash": "771MOg=="},
      {"hash": "5eOrwQ=="}
    ]
  }
}
```

###### Client information

The `clientID` and `clientVersion` fields should uniquely identify a client implementation, not an
individual user. (Client information is used in server-side logging. You can choose any name for the
client ID, but we suggest you choose a name that represents the true identity of the client, such as
your company name, presented as all one word, in all-lowercase letters.)

###### All client states

The `clientStates` field holds the client states for *all* Safe Browsing lists in
the client's local database. (Client states are returned in the `newClientState` field of the
[threatListUpdates.fetch response](https://developers.google.com/safe-browsing/v4/update-api#http-post-response).)

###### Safe Browsing lists

The `threatTypes`, `platformTypes`, and `threatEntryTypes`
fields combine to identify (name) the Safe
Browsing lists. In the example, two lists are identified: MALWARE/WINDOWS/URL and
SOCIAL_ENGINEERING/WINDOWS/URL. Before sending a request, make sure the type combinations you
specify are valid (see [Safe Browsing Lists](https://developers.google.com/safe-browsing/v4/lists)).


###### Threat hash prefixes

The threatEntries array contains the hash prefixes of the URLs that you want to check.
The `hash` field must contain the exact hash prefix that is present in the local database. For
example, if the local hash prefix is 4 bytes long then the threat entry must be 4 bytes long. If the
local hash prefix was lengthened to 7 bytes then the threat entry must be 7 bytes long.

In the example, the request includes three hash prefixes. All three prefixes will be compared
against each of the Safe Browsing lists to determine if there is a matching full-length hash.

**Note:** The Update API and the fullHashes.find method should always use the `hash` field,
never the `URL`field (see [ThreatEntry](https://developers.google.com/safe-browsing/reference/rest/v4/ThreatEntry)).

### HTTP POST response

In the following example, the response returns the matching data, organized by Safe Browsing list,
along with the cache and wait durations.

#### Response header

The response header includes the [HTTP status code](https://developers.google.com/safe-browsing/v4/status-codes)
and the content type. Clients that receive a status code other than HTTP/200 must back-off (see
[Request Frequency](https://developers.google.com/safe-browsing/v4/request-frequency)).

```
HTTP/1.1 200 OK
Content-Type: application/json
```

#### Response body


The response body includes the match information (the list names and the full length hashes, the
metadata, if available, and the cache durations). In the example, the response body also includes a
minimum wait duration. For more details, see the
[fullHashes.find response body](https://developers.google.com/safe-browsing/reference/rest/v4/fullHashes/find#response-body)
and the explanations that follow the code example.

```
{
  "matches": [{
    "threatType":      "MALWARE",
    "platformType":    "WINDOWS",
    "threatEntryType": "URL",
    "threat": {
      "hash": "WwuJdQx48jP-4lxr4y2Sj82AWoxUVcIRDSk1PC9Rf-4="
    },
    "threatEntryMetadata": {
      "entries": [{
        "key": "bWFsd2FyZV90aHJlYXRfdHlwZQ==",  // base64-encoded "malware_threat_type"
        "value": "TEFORElORw=="  // base64-encoded "LANDING"
       }]
    },
    "cacheDuration": "300.000s"
  }, {
    "threatType":      "SOCIAL_ENGINEERING",
    "platformType":    "WINDOWS",
    "threatEntryType": "URL",
    "threat": {
      "hash": "771MOrRPMn6xPKlCrXx_CrR-wmCk0LgFFoSgGy7zUiA="
    },
    "threatEntryMetadata": {
      "entries": []
    },
    "cacheDuration": "300.000s"
  }],
  "minimumWaitDuration": "300.000s",
  "negativeCacheDuration": "300.000s"
}
```

###### Matches

The Matches object returns a matching full-length hash for two of the hash prefixes. The URLs
corresponding to these hashes are considered unsafe. No match was found for the third hash prefix,
so nothing is returned; the URL corresponding to this hash prefix is considered safe.

Note that this example matches one full-length hash to one hash prefix; there could, however, be
multiple full hashes that map to the same hash prefix.

###### Metadata

The `threatEntryMetadata` field is optional and provides additional information about the threat
match. Currently, metadata is available for the MALWARE/WINDOWS/URL Safe Browsing list
(see [Metadata](https://developers.google.com/safe-browsing/v4/metadata)).

###### Cache durations

The `cacheDuration` and `negativeCacheDuration` fields indicate the amount
of time the hashes must be
considered either unsafe or safe (see [Caching](https://developers.google.com/safe-browsing/v4/caching)).

###### Minimum wait durations

The `minimumWaitDuration` field indicates that the client must wait 300 seconds (5 minutes) before
sending another fullHashes request. Note that a wait period may or may not be included in the response
(see [Request Frequency](https://developers.google.com/safe-browsing/v4/request-frequency)).