Limits and quotas

  • Google Cloud projects used with the Data Manager API have limits of 100,000 requests per day and 300 requests per minute.

  • Exceeding project limits results in RESOURCE_EXHAUSTED errors and HTTP status 429 Too Many Requests.

  • Specific request types like IngestAudienceMembersRequest and IngestEventsRequest have limits on the number of resources and user identifiers they can contain.

Follow this guide to understand limits and quotas when working with the Data Manager API.

Project limits

You need a Google Cloud project in order to use the Data Manager API. The Google Cloud project provides the OAuth client and credentials that you use to authenticate API requests.

The Data Manager API enforces rate limits on each Google Cloud project, separated by service and operation type (read versus mutate):

Service Operation Requests per day Requests per minute
IngestionService Mutate (write) 100,000 300
IngestionService Read 100,000 300
All other services combined Mutate (write) 50,000 300
All other services combined Read 50,000 300

Requests that exceed the limits are rejected with the error RESOURCE_EXHAUSTED and HTTP status 429 Too Many Requests.

You can take control of your app and mitigate rate limits by both actively reducing the number of requests and throttling QPS from the client side. You can also set up quota alerts and monitoring in the Google Cloud console to receive notifications before your application reaches its limits.

Quota names

Each limit corresponds to a quota limit name and metric in Google Cloud. When a request exceeds a limit, the Data Manager API returns the quota limit name in the ErrorInfo standard detail payload under metadata.quota_limit:

Quota limit name Service Operation Window Limit
IngestionMutateRequestsPerDayPerProject IngestionService Mutate Day 100,000
IngestionMutateRequestsPerMinutePerProject IngestionService Mutate Minute 300
IngestionReadRequestsPerDayPerProject IngestionService Read Day 100,000
IngestionReadRequestsPerMinutePerProject IngestionService Read Minute 300
ManagementMutateRequestsPerDayPerProject All other services Mutate Day 50,000
ManagementMutateRequestsPerMinutePerProject All other services Mutate Minute 300
ManagementReadRequestsPerDayPerProject All other services Read Day 50,000
ManagementReadRequestsPerMinutePerProject All other services Read Minute 300

Inspect metadata.quota_limit in the ErrorInfo payload to distinguish between daily and per-minute limit errors and determine whether to pause temporarily and retry with backoff, or pause until the daily quota resets. For details, see Quota and rate limit errors.

Handle rate limits and concurrency

To maximize throughput without exceeding project quotas, follow these guidelines:

  • Batch records per request: Include as many records as possible in each request up to the request limits (up to 10,000 AudienceMember resources or 2,000 Event resources per request).

  • Send up to 10 concurrent requests: To reduce total ingestion time without exceeding the 300 requests per minute limit, send up to 10 concurrent requests to IngestionService.

  • Retry per-minute rate limits with backoff: If a request fails with RESOURCE_EXHAUSTED (HTTP 429) and metadata.quota_limit indicates a per-minute limit (such as IngestionMutateRequestsPerMinutePerProject), retry using exponential backoff with jitter (for example, waiting 1s, 2s, and 4s between retries).

  • Pause on daily quota exhaustion: If metadata.quota_limit indicates a daily limit (such as IngestionMutateRequestsPerDayPerProject), don't retry immediately. Pause requests until the daily quota resets at midnight Pacific Time.

Request limits

Here are the limits for specific requests: