Load Balancing for Open Bidder

Bidder deployments may need load balancing if they have more than a single bidder per zone—either for serving more traffic than a single instance can handle, or for high availability. Open Bidder's load balancing is supported through Google Compute Engine, which offers global load balancing for traffic serviced by bidder instances.

There are two types of load balancing: Network and HTTP. Either can be used, though HTTP load balancing is ideal for bidders since they only serve HTTP or HTTPS requests.

HTTP/HTTPS load balancing

HTTP load balancing is very flexible and supports many configuration options. Bidders don't typically need sophisticated balancing setups such as cross-region balancing, since Ad Exchange already has custom routing from its trading regions to Google Compute Engine zones. Also since bidders only serve a small variety of HTTP requests and don't need CDNs, proxies, caches, etc., content-based load balancing is unnecessary.

HTTP load balancing requires manual setup of several managed resources. We recommend using a simple structure of Google Compute Engine resources as explained below. You will configure these resources manually or via scripts using gcloud compute, or visually via Google API Console > Networking > Load Balancing > HTTP load balancing . The following instructions (based on the Cross-Region Load Balancing example) are for the command line option; just replace the names of network, region, zone and instance resources for your project.

Set up HTTP load balancing

Here's how we'll map Open Bidder entities to the Google Compute Engine resources:

  • A single, project-wide health check will point to the bidders' public HTTP Port.
  • Each bidder zone needs one instance group containing its bidders, and one backend service connected to that instance group. These resources are always 1:1 because we don't need cross-region balancing. Use balancing mode is by RATE and have all bidders in the group created with the same machine type; this produces best latency distribution.
  • Each zone will also have one URL map (default configuration since we don't need any content-based balancing); one target proxy, and one forwarding rule (always --global even though we'll create one per-zone to take advantage of the low-latency Ad Exchange/Google Compute Engine routing).
  • Forwarding rules can use an ephemeral IP for testing with HTTP, but a static IP address resource is recommended for production, and it's always required with HTTPS balancing.

Here's a script that creates all these resources, for a sample test deployment using one zone with two bidders, default port configuration, and a max of 20k requests per second per Compute Engine virtual machine instance in the instance group:

Note: The 20k max rate per instance specified for the load balancers here is not the same as the QPS quota specified in Ad Exchange's callout proxy / rtb-keys configuration. The max rate per instance should be greater than or equal to Ad Exchange's callout proxy quota to allow Ad Exchange to control traffic throttling downstream.

Create HTTP balancing resources

ZONE=us-central1-a
NETWORK=network-1
BIDDERS="bidder-1 bidder-2"
gcloud compute addresses create bidip-1 --global
IP_ADDR=`gcloud compute addresses describe bidip-1 --global --format yaml | \
         grep "address:" | cut -f 2 -d ' '`
gcloud compute http-health-checks create bidhc --port 80 --request-path /admin/healthcheck
gcloud compute instance-groups unmanaged create bidgroup-1 --zone $ZONE
gcloud compute instance-groups unmanaged add-instances bidgroup-1 \
       --instances $BIDDERS --zone $ZONE
gcloud compute instance-groups unmanaged set-named-ports bidgroup-1 \
       --named-ports http:80 --zone $ZONE
gcloud compute backend-services create bidservice-1 --http-health-check bidhc --port 80
gcloud compute backend-services add-backend bidservice-1 --balancing-mode RATE \
       --max-rate-per-instance 20000 --zone $ZONE --instance-group bidgroup-1
gcloud compute url-maps create bidurl-1 --default-service bidservice-1
gcloud compute target-http-proxies create bidproxy-1 --url-map bidurl-1
gcloud compute forwarding-rules create bidfr-1 --address $IP_ADDR --global \
       --target-http-proxy bidproxy-1 --port-range 80

After creating your load balancing resources, you need to edit your DNS records for your domain. You must create a new A record to map your zone's hostname to the IP address of your load balancing forwarding rule. Upon setup completion, your bidders should start to receive requests from Ad Exchange.

Notice that a traceroute into your bidders' hostnames will terminate in their forwarding rules' addresses. You can check if specific bidders are receiving requests via the bidder's metrics page or SSH, or with API Console > Compute > Compute Engine > VM instances > Network traffic.

Use gcloud compute backend-services get-health bidservice-1 to check the health of all bidders. If some bidder is not healthy, you can query its healthcheck request path (from the API Console, curl, etc.—notice you need the instance's public IP and port, not the balancer's). If a health check fails, this can mean the bidder service has not yet started (which may take a few minutes after the instance appears as RUNNING), or perhaps its startup has failed (typically due to bad parameters); to know what's wrong you can check the bidder logs in the API Console's Monitoring > Logs.

If you later delete or create other bidder instances in this zone, don't forget to update the instance groups (with gcloud compute target-pools or the API Console). Use gcloud compute instances to automatically discover the active bidders in a given zone, so these load balancing maintenance tasks are not hard to script out.

When you stop bidding on this zone, do not forget to clean up its load balancing resources.

Secure bidding with the HTTPS load balancer

Starting in 2015, Ad Exchange is moving its inventory to require encryption everywhere, so bidders should now support HTTPS requests. Here's a new script to create the Google Compute Engine HTTPS load balancing resources, updated from the previous example:

Create HTTPS balancing resources

ZONE=us-central1-a
NETWORK=network-1
BIDDERS="bidder-1 bidder-2"
gcloud compute ssl-certificates create bidcert-1 \
       --certificate example.crt --private-key example.key
gcloud compute addresses create bidip-1 --global
IP_ADDR=`gcloud compute addresses describe bidip-1 --global --format yaml | \
         grep "address:" | cut -f 2 -d ' '`
gcloud compute https-health-checks create bidhc --port 443 --request-path /admin/healthcheck
gcloud compute instance-groups unmanaged create bidgroup-1 --zone $ZONE
gcloud compute instance-groups unmanaged add-instances bidgroup-1 \
       --instances $BIDDERS --zone $ZONE
gcloud compute instance-groups unmanaged set-named-ports bidgroup-1 \
       --named-ports https:443 --zone $ZONE
gcloud compute backend-services create bidservice-1 --protocol HTTPS \
       --https-health-check bidhc --port 443
gcloud compute backend-services add-backend bidservice-1 --balancing-mode RATE \
       --max-rate-per-instance 20000 --zone $ZONE --instance-group bidgroup-1
gcloud compute url-maps create bidurl-1 --default-service bidservice-1
gcloud compute target-https-proxies create bidproxy-1 --url-map bidurl-1 \
       --ssl-certificate bidcert-1
gcloud compute forwarding-rules create bidfr-1 --address $IP_ADDR --global \
       --target-https-proxy bidproxy-1 --port-range 443

You can set up DNS as before, and the bidders will start accepting requests (bid requests, impressions, etc.) via HTTPS. A wildcard certificate is recommended so that you can create a bunch of bidders as subdomains (e.g. bidder1.myads.com, bidder2.myads.com etc.) and still use the same certificate, even if you need multiple zones and proxies.

Open Bidder's load balancing support has the following limitations:

  • The Netty webserver stack handles SSL with a self-signed certificate that's created automatically at startup. This is OK because Google Compute Engine load balancing doesn't require certificate validation from the destination instances. However, you can't receive HTTPS requests directly from Ad Exchange without first going through the load balancer.
  • The Jetty webserver stack doesn't support SSL at all; we have no plans to change that. Starting with 0.9.0, the Jetty stack is definitely demoted to a "reference" web stack, not intended for RTB requests in production.

Load Balancing with HTTPS + HTTP

A third, hybrid solution is using HTTPS in the exchange→balancer path, but then using plain HTTP for the balancer→bidder path. The tradeoff in security is small, since this last hop is protected from most attacks by the network setup (well, as long as your bidder network's firewall is closed tight and the bidder server not compromised!). We recommend evaluating if the CPU saving in bidders is worth this tradeoff.

Here's how you can enable this configuration: In the UI, Network, set the HTTP Port, but unset the HTTPS Port (empty field or -1). Then use the "hybrid" script below.

That's it! HTTPS will enter the balancer by the SSL-enabled Forwarding Rule on port 443, then sent to a Target HTTPS Proxy where SSL termination happens. The proxy's URL map routes the request into a backend service in the same port; but now this resource is configured as HTTP, so the request follows the rest of the way to the bidder in plain HTTP.

Send feedback about...

Open Bidder (Beta)