Local Databases

This document applies to the following method: Update API (v4): threatListUpdates.fetch.

Database setup

Clients using the Update API are required to set up a local database and to perform an initial download of the Safe Browsing lists they want to work with. To get going, you can build and deploy the safebrowsing Go package (or use the package to model your own implementation). For more information see https://github.com/google/safebrowsing/.

Database updates

To ensure protection against the latest threats, clients are strongly encouraged to regularly update their local Safe Browsing lists using the threatListUpdates.fetch method. The threatListUpdates.fetch request specifies the lists to be updated. If clients have memory or bandwidth limitations, they can also use the request to set update constraints (see Update Constraints). The threatListUpdates.fetch response returns either a full update or partial update for each list, as explained below.

Full updates

Full updates are returned when the client leaves the state field in the threatListUpdates.fetch request empty or when the server determines a full update is required. For full updates, only additions are returned. The client is expected to clear the local database before applying the updates and performing the validation check.

Empty state

Full updates are returned when the client sends the initial request for a list. In this case, the state field in the request is left empty (because there is no value to provide) and the newClientState field in the response returns the initial state for the local list. Full updates are also returned when the client purposely leaves the state field empty on subsequent requests. This will force a full update and return a new state in the newClientState field of the response.

Server decision

Occasionally, the Safe Browsing server returns a full update when only a partial update was requested by the client. This may happen when the client initially downloads a small version of the list and then updates to a larger version of the list; the server will simply return a full update with the entire list. This may also happen if the client hasn't downloaded data in a long time and requests a partial update; again, the server will simply return a full update with the entire list.

Partial updates

Partial updates are returned when the client supplies a value for the state field in the threatListUpdates.fetch request (the exception, as noted above, is when the server determines a full update is required). For partial updates, both additions and removals are returned. The client updates the lists in the local database (applying the removals before the additions) and then performs the validation check.

Additions

Additions are SHA256 hash prefixes that should be added to the local database. The vast majority of hash prefixes are 4 bytes long but some hash prefixes may have any length between 4 and 32 bytes. Therefore, multiple sets of additions could be returned; for example, one containing the 4-byte prefixes and one containing 5-byte prefixes.

If the client supports compression, the response may be compressed using Rice compression. However, only 4-byte hash prefixes get compressed. Longer hash prefixes are always sent in uncompressed, raw format (see Compression).

Removals

Removals are zero-based indices in the lexicographically-sorted client database pointing at entries that should be removed from the local database. Only one set of removals will be returned.

If the client supports compression, "rice hashes" and "rice indices" are returned. If compression is not supported, "raw hashes" and "raw indices" are returned (see Compression).

Validation checks

When the threatListUpdates.fetch response is returned—with either a full update or a partial update—the client is expected to perform a validation check.

The client first updates the lists in the local database (applying the removals before the additions). The client then computes the SHA256 hash of the (lexicographically sorted) local list and compares it to the checksum returned in the response. If the two values are equal, the Safe Browsing list is considered "correct."

If the two values are not equal, the Safe Browsing list is considered "corrupt." The client must clear the list from the database and reissue a second update with the state field set to the empty string; this will force a full update and return a brand new list and state.