Use the Population Dynamics Insights embeddings

Prepare ground truth data

To use Population Dynamics embeddings, your ground truth data must be aggregated to a supported geographic boundary. Because administrative boundary types vary globally, you can align your data using either universal mathematical grid systems (like S2 cells) or local administrative regions (such as counties or districts, depending on the specific country dataset).

Option 1: Incorporate embeddings into an existing model

  • Prepare existing model-based ground truth: Use the embeddings as geospatial covariates to enhance an existing model.
  • Train an error correction model: Improve an existing model by integrating the embeddings into a model that takes the original model output, the expected value or ground truth, and the embeddings to learn a new error correction model.

Option 2: Tune for specific use cases

  • Choose a prediction model: Any model, such as GBDT, MLP, or linear, can be used for predictions.
  • Use embeddings for prediction: Use Population Dynamics embeddings as input features, alongside other contextual data, to improve prediction accuracy.

Custom boundary aggregation

If your ground truth data uses custom polygons, such as postal codes, drive-time isochrones or trade areas, you can perform a boundary aggregation. This process combines multiple S2 cell vectors into a single representation for the target polygon. Choosing the right weighting methodology ensures the aggregated embedding accurately reflects your downstream modeling goals.

Use population-weighted aggregation for human-centric use cases, such as retail store performance or consumer behavior modeling.

Using an area-weighted spatial aggregation for demographic data can warp your machine learning features. This occurs when unpopulated areas, such as parks, industrial zones, or bodies of water, distort the profile of actual residents.

To resolve this, you can perform a population-weighted average in BigQuery. This approach uses high-resolution demographic datasets, such as WorldPop in the Earth Engine Data Catalog, to calculate the precise density of each intersecting S2 cell segment.

To see a complete implementation example of a population-weighted workflow, run the interactive notebook.

Colab logo Run in Google Colab GitHub logo View source on GitHub

2. Area-weighted average

For environmental or physical use cases, use area-weighted aggregation instead. This is useful for land use analysis, built environment studies, or infrastructure planning where you must assess regions regardless of population distribution.

In these scenarios, physical land area is more relevant than human population density. This makes sure every square kilometer within the polygon boundary contributes equally to the aggregated vector.

Under this method, each constituent S2 cell's embedding vector is weighted by the geographic surface area it covers within the target polygon.

Query examples

Replace your-project.your_dataset.embeddings_table with your actual project, dataset, and target table name.

SQL: Fetch embeddings

This query retrieves the embedding vector and administrative metadata for the S2 cells in your provisioned dataset.

SELECT
  geo_id,
  administrative_area_level_1_name AS state,
  administrative_area_level_2_name AS county,
  features -- The 330-dim vector
FROM
  `your-project.your_dataset.embeddings_table`
LIMIT 10;

SQL: Find similar locations

This query identifies behaviorally similar locations without requiring external data.

It uses the ML.DISTANCE function to calculate cosine similarity, returning the top matches for a target S2 cell. This approach supports expansion planning scenarios, such as determining where to open a new store based on the profile of a successful existing location.

To visualize S2 cells on a map, you must convert or join the S2 cell ID to its corresponding polygon geometry, because this dataset uses S2 cell tokens instead of latitude and longitude points.

WITH TargetLocation AS (
  SELECT features AS target_vector
  FROM `your-project.your_dataset.embeddings_table`
  -- Replace with your target S2 hex token (e.g., '80ead45')
  WHERE geo_id = 'YOUR_TARGET_S2_TOKEN'
)

SELECT
  t.geo_id,
  t.administrative_area_level_1_name AS state,
  t.administrative_area_level_2_name AS county,
  -- Calculate Similarity (1.0 is identical, 0.0 is dissimilar)
  (1 - ML.DISTANCE(t.features, p.target_vector, 'COSINE')) AS similarity_score
FROM
  `your-project.your_dataset.embeddings_table` t,
  TargetLocation p
WHERE
  t.geo_id != 'YOUR_TARGET_S2_TOKEN' -- Exclude the target itself
ORDER BY
  similarity_score DESC
LIMIT 20;

SQL: Join customer data

This example demonstrates how to enrich your own internal data (for instance, a store performance table) with behavioral embeddings. Ensure your internal data includes matching S2 cell tokens (hex strings).

SELECT
  store.store_id,
  store.s2_token,
  store.total_revenue,
  embeddings.features AS pdfm_vector
FROM
  `your-project.internal_data.store_performance` AS store
JOIN
  `your-project.your_dataset.embeddings_table` AS embeddings
ON
  -- Join based on the S2 hex token string
  store.s2_token = embeddings.geo_id

Python: Load data for machine learning

The embeddings are stored as BigQuery Arrays. To use them in ML libraries, you must convert the column into a NumPy matrix.

from google.cloud import bigquery
import numpy as np
import pandas as pd

client = bigquery.Client()

query = """
    SELECT
        geo_id,
        features -- Returns as a list of floats
    FROM
        `your-project.your_dataset.embeddings_table`
    LIMIT 1000
"""

# 1. Load data into DataFrame
df = client.query(query).to_dataframe()

# 2. Convert the 'features' column (Series of Lists) into a Matrix (2D Array)
X_matrix = np.stack(df['features'].values)

print(f"Data Loaded. Matrix Shape: {X_matrix.shape}")
# Output: Data Loaded. Matrix Shape: (1000, 330)