Allow Google-Agent traffic
Google-Agent
is the name used by agents hosted on Google infrastructure to navigate the web and perform actions
upon user request. Explicitly allowing Google-Agent traffic in your content delivery
network (CDN), web application firewall (WAF), or bot-detection service helps prevent legitimate
requests from being blocked while preserving your existing security controls.
How Google-Agent identifies requests
Each Google-Agent request includes a User-Agent HTTP request header
containing "Google-Agent" (see
Google user-triggered fetchers)
and comes from an IP address from
user-triggered-agents.json.
Google is also experimenting with the
Web Bot Auth protocol using the
https://agent.bot.goog identity.
In this experiment, Google signs requests made by Google-Agent
with the HTTP Message Signatures standard
(RFC 9421).
Participating requests include:
- A
Signature-Agentheader set tog="https://agent.bot.goog" SignatureandSignature-Inputheaders labeled asg
The public key set for verifying these signatures is available at: https://agent.bot.goog/.well-known/http-message-signatures-directory
Allow traffic in your CDN, Firewall, or bot-detection service
Some bot-detection services, CDNs, and Firewalls provide ways to verify requests made by the
Google-Agent user agent and allow these requests; refer to your provider for the
exact steps.
If your service doesn't automatically verify Google-Agent requests, you may be able
to verify and allow requests using the following rules:
- Confirm the
User-Agentheader containsGoogle-Agent. - Confirm the IP address as being in user-triggered-agents.json.
-
If provided, confirm the signed request by verifying the
SignatureandSignature-Inputheaders forSignature-Agent: g="https://agent.bot.goog"according to Web Bot Auth. - For verified requests, skip bot challenges.
Troubleshooting
If legitimate Google-Agent requests are still blocked after allowing these requests:
- Rule priority: Confirm that your allow rule has higher priority than (and evaluates before) broad cloud or data center ASN blocks, managed bot rules, or category-level AI agent block policies that could override a verified bot setting.
- Interactive challenges: Confirm that your rule skips interactive CAPTCHAs and JavaScript challenges in addition to HTTP blocks, including standalone in-page CAPTCHA widgets triggered on search or form submission.
-
In-journey actions and APIs: Confirm that allowing these requests covers actions beyond
initial homepage loads, including on-site search, filtering, clicking into detail pages,
add-to-cart, booking, and underlying XHR, GraphQL, or
api.*endpoints. -
Shared-IP rate limits: Confirm that per-IP rate-limiting rules account for or exempt
verified
Google-AgentIP ranges, as concurrent requests from multiple users share the same egress pool. - IP range updates: Confirm that IP-based rules stay synced with the latest ranges in user-triggered-agents.json.
-
Proxy headers: Confirm that load balancers and intermediate proxies preserve the original
client IP, the
User-Agentheader, and (for signed requests) theSignature,Signature-Input, andSignature-Agentheaders.