# Precomputed Embeddings for Imaging Data Commons (IDC) Pathology Imaging

April 6th, 2026, Kenneth Philbrick PhD   

HealthAI, Google Research

![Illustration of Imaging Data Commons website](https://developers.google.com/static/health-ai-developer-foundations/blogs/idc_image_search/white_paper_images/idc.jpg)

[Imaging Data Commons](https://imaging.datacommons.cancer.gov) (IDC) is one of
the largest publicly available, de-identified, repositories for cancer imaging.
The repository is funded by the National Cancer Institute (NCI), an institute of
the National Institutes of Health (NIH), a part of the US Department of Health
and Human Services. IDC contains imaging for all major medical imaging
modalities. Imaging is stored within the archive as
[DICOM](https://www.dicomstandard.org/). Imaging and its associated metadata can
be searched, visualized through the [IDC
website](https://portal.imaging.datacommons.cancer.gov/explore/),
[BigQuery](https://cloud.google.com/healthcare-api/docs/resources/public-datasets/idc),
and can be accessed using DICOMweb (IDC,
[tutorial](https://learn.canceridc.dev/data/downloading-data/dicomweb-access)).

Pathology imaging is considered the gold standard for many cancer diagnoses. To
reduce the barrier to the development of novel ML models from the IDC-hosted
imaging, we have pre-computed [Path
Foundation](https://developers.google.com/health-ai-developer-foundations/path-foundation)
embeddings for the majority of the IDC pathology imaging.

[Path Foundation](https://developers.google.com/health-ai-developer-foundations/path-foundation)
is a patch-based, domain-specific, foundation model. The model was trained using
[SSL techniques](https://arxiv.org/abs/2310.13259). The model generates
embeddings, 384 floats, for input images that are 224 x 224 pixels. The model is
available on [Hugging Face](https://huggingface.co/google/path-foundation) (open
weight) and on [Model
Garden](https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/path-foundation).

## Generation Precomputed Embeddings

IDC version 20, March, 2025, contains about 68 thousand whole slide microscopy
images. Precomputed embeddings have been generated for all but a handful of the
IDC imaging.

Path Foundation embeddings were computed by sampling patches from the
tissue-containing regions of slide imaging. Tissue regions of the slide were
determined by thresholding low-magnification imaging, \~2X, and determined by
pixels with a HSV value component, 0 \< value \<= 204; range empirically
determined. Non-overlapping 224 x 224 pixel patches were sampled from
high-resolution imaging (\~20x or greater) and embeddings were computed for the
sampled patches.

The number of embeddings generated per-slide range from a few hundred embeddings
to well over 100 thousand embeddings. For slides with more than 500 embeddings,
embeddings were clustered using [scikit-learn
MiniBatchKMeans](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.MiniBatchKMeans.html)
with parameters (n_clusters=500, max_iter=600, n_init=10, batch_size=10240) to
generate a reduced representation of the slide's patch embeddings.

K-Means is a clustering technique used here to reduce the large set of
embeddings to a smaller representative set of embedding centroids. To see an
example of this approach you can view the [HAI-DEF outlier detector
demo](https://huggingface.co/spaces/google/path-foundation-demo), which clusters
embeddings of non-tumor tissues for whole-slide imaging from the [Camelyon data
challenge](https://camelyon17.grand-challenge.org/) to create a model for
outlier detection.

For each slide, all generated data is available on [Google Cloud
Storage](https://cloud.google.com/storage) (GCS).
[BigQuery](https://cloud.google.com/bigquery) tables were created to join IDC
BigQuery tables with GCS resources using joins performed using either the source
imaging IDC UUID or DICOM StudyInstanceUID, SeriesInstanceUID, and
SOPInstanceUID. Additionally, a BigQuery table has been created that hosts the
k-means clustered embeddings. To accelerate search, this data has been indexed
using the [BigQuery IVF
method](https://docs.cloud.google.com/bigquery/docs/vector-index).

![Representation k-means clustering of patch embeddings sampled across a whole
slide image](https://developers.google.com/static/health-ai-developer-foundations/blogs/idc_image_search/white_paper_images/image_patches.jpg)

**Figure 1**: Provides a conceptual illustration of the patch sampling used to
generate the pathology embeddings. Each colored square in the image on the right
represents a patch sampled for embedding generation. The color of the
rectangular patch conceptually represents the index of the k-means generated
embedding centroid which was closest to the patch embedding.

### **Pre-computed Embedding Datasets:**

#### BigQuery Embedding Dataset

**GCP Project** : hai-cd3-foundations, **Dataset** : idc_metadata   

**Dataset License** : [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/deed.en)

- **Resource Table** (Table:
  [pathology_embeddings_metadata](https://console.cloud.google.com/bigquery?project=hai-cd3-foundations&ws=!1m10!1m4!4m3!1shai-cd3-foundations!2sidc_metadata!3spath_foundation_500_embeddings!1m4!4m3!1shai-cd3-foundations!2sidc_metadata!3spathology_embeddings_metadata)):
  Defines mapping between HAI-DEF embedding resources on GCS and Imaging Data
  Commons imaging.

- **Search Table** (Table:
  [path_foundation_500_embeddings](https://console.cloud.google.com/bigquery?project=hai-cd3-foundations&ws=!1m5!1m4!4m3!1shai-cd3-foundations!2sidc_metadata!3spath_foundation_500_embeddings)):
  Contains pre-computed, indexed, digital pathology embeddings for embedding
  similarity search BigQuery example.

Additional Dataset:

- [Imaging Data Commons IDC
  Archive](https://docs.cloud.google.com/healthcare-api/docs/resources/public-datasets/idc): IDC generated tables that describe all imaging stored within Imaging Data Commons.

##### Example: Finding Similar Slides Using BigQuery Vector Search

This example illustrates performing whole slide image similarity search across
IDC imaging using BigQuery vector search. To perform whole slide image
similarity search, embeddings are generated across a search slide and stored
within a BigQuery table. These embeddings define the search set. Vector search
is performed against stored and indexed IDC archive embeddings (N = 32,465,939
embeddings) by performing a series of nested SQL queries using BigQuery. The
embedding similarity search is parameterized to return the top 20 results for
each embedding in the search set. Embedding vector similarity is computed using
[cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity).
Per-embedding search results are then aggregated at the slide level to determine
the overall similarity between whole slide images in the IDC archive and the
search set. Slide level aggregation is performed by defining the overall whole
slide image similarity score as the sum of the cosine similarity scores. The sum
of scores works here as an aggregation method because cosine similarity score
metric and embedding similarity are positively correlated. Alternative embedding
search strategies will result in different results. As one example, changing the
distance aggregation metric from `sum(CosineSimilarity)` to
`max(CosineSimilarity)` will prioritize the identification of imaging that
closely matches at least one of the search embeddings but otherwise may be quite
different.

##### Example BigQuery SQL to Perform Similarity Search

    SELECT
    idc_table.CollectionName AS CollectionName,
    idc_table.ProtocolName AS ProtocolName,
    embedding_search.StudyInstanceUID AS StudyInstanceUID,
    embedding_search.SeriesInstanceUID AS SeriesInstanceUID,
    embedding_search.SumCosineSimilarity AS SumCosineSimilarity,
    FROM
    # Get series descriptive metadata from IDC table.
    (
      SELECT DISTINCT
        collection_name AS CollectionName,
        ClinicalTrialProtocolName AS ProtocolName,
        StudyInstanceUID,
        SeriesInstanceUID
      FROM `bigquery-public-data.idc_current.dicom_all`
    ) AS idc_table,
    (
      # Get Sum of found search embedding cos distances
      # per Unique DICOM (StudyInstanceUID, SeriesInstance UID)imaging.
      #
      # Cosine Dist
      # 1 = vector pointing in same direction
      # -1 = vector pointing in opposite direction.
      SELECT
        StudyInstanceUID, SeriesInstanceUID, sum(CosineSimilarity) AS SumCosineSimilarity
      FROM
        (
          # Return Study instance uid, series instance uid, and
          # and distance search embeddings and source..
          SELECT
            base.StudyInstanceUID AS StudyInstanceUID,
            base.SeriesInstanceUID AS SeriesInstanceUID,
            distance AS CosineSimilarity
          FROM
            # Performs a vector search in BQ using COSINE similarity
            # Find images which best match search embeddings listed in:
            # pathology-cloud-dev.idc_test_us.search_embeddings
            VECTOR_SEARCH(
              # Search over embeddings excluding any image from DICOM
              # that the search embeddings are derived from
              (
                SELECT *
                FROM `hai-cd3-foundations.idc_metadata.path_foundation_500_embeddings`
                WHERE
                  # Exclude any imaging from the same DICOM study that the search
                  # embeddings were generated. This where clause can be omitted entirely
                  # if novel imaging is used. If imaging from another IDC study is used
                  # to perform the search then the StudyInstanceUID should be updated to
                  # match the values used to define the embedding search (Part 1).
                  StudyInstanceUID != '2.25.48791557373299768401597362411459861639'
              ),
              'embedding',
              TABLE `hai-cd3-foundations.idc_metadata.path_foundation_500_search_example`,
              top_k => 20,  # Return 20 closest matches for each search embedding
              distance_type => 'COSINE',  # Embedding search distance metric
              options => '{"fraction_lists_to_search": 0.005}')
        )
      GROUP BY StudyInstanceUID, SeriesInstanceUID
      ORDER BY SumCosineSimilarity DESC
      LIMIT 100  # Limit results to top 100, closest source embeddings
    ) AS embedding_search
    # Join IDC metadata table with embedding search results.
    WHERE
    idc_table.StudyInstanceUID = embedding_search.StudyInstanceUID
    AND idc_table.SeriesInstanceUID = embedding_search.SeriesInstanceUID
    # list results largest first.
    ORDER BY SumCosineSimilarity DESC

**Example BigQuery Search Results** : ![example big query results](https://developers.google.com/static/health-ai-developer-foundations/blogs/idc_image_search/white_paper_images/bq_results.jpeg)

#### Google Cloud Storage (GCS) Embedding Dataset

**Location** : gs://healthai-us/encoded-data/idc/pathology   

**Dataset License** : [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/deed.en)

Data for each slide is written to GCS to the following organizational structure:


gs://healthai-us/encoded-data/idc/pathology/**{imaging_idc_uuid}**/path_foundation/v1/...

**Overview of Dataset**:

- **Metadata** (metadata.json): Imaging and metadata representing the original imaging and sampled areas.
- **Images**

  - slide_thumbnail.jpeg: Thumbnail of whole slide image.

  Example image:

  ![thumbnail image of whole slide image](https://developers.google.com/static/health-ai-developer-foundations/blogs/idc_image_search/white_paper_images/thumbnail.jpg)
  - slide_thumbnail_mask.jpeg: Binary mask projected over thumbnail image illustrating regions of a WSI image sampled for embedding generation.

  Example image:

  ![mask illustrating where patches were sampled for embedding
  generation.](https://developers.google.com/static/health-ai-developer-foundations/blogs/idc_image_search/white_paper_images/mask.jpg)
- **Path Foundation Embeddings** (slide_embedding.npy): Patch-level embeddings
  generated for each slide.

- **Coordinates** (slide_embedding_coord.npy): Location (pixels) in source
  imaging of left coordinate where each patch was sampled for embedding
  generation.

- **K-Means Representations** (embeddings_500_cluster_centroid.npy): The
  k-means reduced representation of each slide's patch embeddings.

- **Index Mapping** (embeddings_500_labels.npy): Maps patch embeddings to
  their closest k-means embedding.

**Detailed description of metadata.json**:

| JSON Key | Description |
|---|---|
| instance_uuid | [IDC instance uuid](https://learn.canceridc.dev/data/organization-of-data/guids-and-uuids) |
| instance_hash | [IDC generated hash of instance bytes](https://learn.canceridc.dev/data/organization-of-data/bigquery-tables#auxiliary_metadata) |
| instance_pixeldata_SHA512_hash | Hash of instance pixel data from which embeddings were generated |
| study_instance_uid | DICOM [StudyInstanceUID](https://learn.canceridc.dev/data/data-model) for the imaging used to generate embeddings |
| series_instance_uid | DICOM [SeriesInstanceUID](https://learn.canceridc.dev/data/data-model) for the imaging used to generate embeddings |
| sop_instance_uid | DICOM [SOPInstanceUID](https://learn.canceridc.dev/data/data-model) for the imaging used to generate embeddings |
| instance_patch_embeddings | Sequence of datasets describing patch embedding datasets |
| \> model_name | Name of HAI-DEF foundation model used to generate embeddings |
| \> model_version | Version of HAI-DEF foundation model used to generate embeddings |
| \> version | Embedding dataset version |
| \> embedding_count | Number of embeddings generated for the slide |
| \> input_image_patch_width_px | Width in pixels of the image patch used to generate embeddings |
| \> input_image_patch_height_px | Height in pixels of the image patch used to generate embeddings |
| \> patch_embeddings_gs_path | GS path to binary numpy file encoding the slide patch embeddings. The shape of the encoded array is \[embedding_count, 384\] |
| \> patch_coordinates_gs_path | GS path to binary numpy file encoding the slide patch coordinates. The coordinates describe the upper left (X, Y) pixel coordinate of the patch. Coordinates are in reference to the patch source image. The dimensions of the source image are encoded in slide_pyramid_level_width and slide_pyramid_level_height keys. The shape of the encoded array is \[embedding_count, 2\]. The patch embedding and coordinates arrays are aligned, such that the embeddings described at: patch_embeddings\[N\] were generated from a patch sampled at patch_coordinates\[N\]. |
| \> thumbnail_patch_sampling_mask_gs_path | GS path to a JPEG image that illustrates where within the image embedding patches were sampled. Image generated by rendering the thumbnail imaging and drawing black squares over the regions sampled by patches. |
| \> thumbnail_gs_path | GS path to a JPEG image that illustrates a low resolution image of the tissue imaging. |
| \> kmeans_embeddings | Sequence of metadata describing k-means reduced embeddings generated from patch embeddings. |
| \>\> number_of_centroids | Number of k-means centroids generated. |
| \>\> centroids_gs_path | GS path to a numpy file encoding the embedding centroids computed by the minibatch_kmeans. The shape of the encoded array is \[number of clusters, 384\]. |
| \>\> embedding_centroid_index_gs_path | GS path to a numpy file encoding the index of the embedding centroid that each of the slides embeddings is nearest to using euclidean distance metric. The shape of the encoded array is \[embedding_count\]. |

##### Programmatically Accessing the GCS Data using Python

Boilerplate code to convert IDC imaging DICOM StudyInstanceUID and
SeriesInstanceUID to IDC instance_uuid.

    from google.cloud import bigquery
    import google.auth
    import requests

    def _get_idc_uuid(study_instance_uid: str, series_instance_uid: str) -> str:
      """Utility function to convert DICOM study and series UID to IDC UUID."""
      credentials = google.auth.default()[0]
      credentials.refresh(google.auth.transport.requests.Request())
      # Perform a query.
      QUERY = (f'SELECT instance_uuid FROM `hai-cd3-foundations.idc_metadata.pathology_embeddings_metadata` WHERE `StudyInstanceUID` = "{study_instance_uid}" AND `SeriesInstanceUID` = "{series_instance_uid}" LIMIT 1')
      client = bigquery.Client(credentials=credentials)
      query_job = client.query(QUERY)  # API request
      results = list(query_job.result())
      if not results:
        raise ValueError('DICOM instance not found.')
      return results[0]['instance_uuid']

**Example 1: Read a Slide embedding metadata.json from GCS**

    import json
    import google.cloud.storage as gcs_storage

    # Slide is identified by StudyInstanceUID and SeriesInstanceUID
    study_instance_uid = '2.25.100000270731860872503075508795340993296'
    series_instance_uid = '1.3.6.1.4.1.5962.99.1.1266341113.500336493.1714958324985.4.0'

    idc_uuid = _get_idc_uuid(study_instance_uid, series_instance_uid)
    client = gcs_storage.Client.create_anonymous_client()
    bucket = gcs_storage.Bucket(name='healthai-us', client=client)
    blob = bucket.get_blob(f'encoded-data/idc/pathology/{idc_uuid}/path_foundation/v1/metadata.json')
    json_str = blob.download_as_string()
    print(json.loads(json_str))

**Example 2: Read a Slide patch embeddings (numpy file) from GCS**

    import io

    import google.cloud.storage as gcs_storage
    import numpy as np

    # Slide is identified by StudyInstanceUID and SeriesInstanceUID
    study_instance_uid = '2.25.100000270731860872503075508795340993296'
    series_instance_uid = '1.3.6.1.4.1.5962.99.1.1266341113.500336493.1714958324985.4.0'

    idc_uuid = _get_idc_uuid(study_instance_uid, series_instance_uid)
    client = gcs_storage.Client.create_anonymous_client()
    bucket = gcs_storage.Bucket(name='healthai-us', client=client)
    blob = bucket.get_blob(f'encoded-data/idc/pathology/{idc_uuid}/path_foundation/v1/slide_embedding.npy')
    with io.BytesIO() as mem:
     blob.download_to_file(mem)
     _ = mem.seek(0)
     embeddings = np.load(mem)
    print(embeddings.shape)

**Example 3: Read a Slide Thumbnail Image from GCS**

    import google.cloud.storage as gcs_storage

    # Slide is identified by StudyInstanceUID and SeriesInstanceUID
    study_instance_uid = '2.25.100000270731860872503075508795340993296'
    series_instance_uid = '1.3.6.1.4.1.5962.99.1.1266341113.500336493.1714958324985.4.0'

    idc_uuid = _get_idc_uuid(study_instance_uid, series_instance_uid)
    client = gcs_storage.Client.create_anonymous_client()
    bucket = gcs_storage.Bucket(name='healthai-us', client=client)
    blob = bucket.get_blob(f'encoded-data/idc/pathology/{idc_uuid}/path_foundation/v1/slide_thumbnail.jpeg')
    blob.download_to_filename('thumbnail.jpeg')

## Generate Pathology Embeddings From DICOM Imaging

The Python library [EZ-WSI
DICOMweb](https://github.com/GoogleCloudPlatform/EZ-WSI-DICOMweb) simplifies
generating Path Foundation embeddings from the DICOM imaging. Refer to the [IDC
documentation](https://learn.canceridc.dev/data/downloading-data/dicomweb-access)
for information about how to access digital pathology imaging from the IDC
archive using EZ-WSI. The EZ-WSI DICOMweb [embedding generation
CoLab](https://colab.research.google.com/github/GoogleCloudPlatform/EZ-WSI-DICOMweb/blob/main/ez_wsi_dicomweb/documentation/getting_started_with_embeddings.ipynb)
provides detailed examples that illustrate how to use EZ-WSI DICOMweb to
generate Path Foundation embeddings.

**Example: Generate a Path Foundation Embedding from a DICOM Whole Slide Image
using EZ-WSI DICOMweb**

    import cv2
    from ez_wsi_dicomweb import credential_factory
    from ez_wsi_dicomweb import dicom_store
    from ez_wsi_dicomweb import patch_embedding
    from ez_wsi_dicomweb import patch_embedding_endpoints

    # URL for IDC DICOM Store (Requires read access to GCP IDC Store)
    # Request for access at:
    # https://docs.cloud.google.com/healthcare-api/docs/resources/public-datasets/idc
    store_url = 'https://healthcare.googleapis.com/v1/projects/nci-idc-data/locations/us-central1/datasets/idc/dicomStores/idc-store-v20/dicomWeb'
    # Connect to IDC DICOM store using application default credentials
    ds = dicom_store.DicomStore(store_url, credential_factory.DefaultCredentialFactory())
    # Study Instance UID and Series Instance UID of imaging.
    study_instance_uid = '2.25.210286627285813284217395096048268203294'
    series_instance_uid = '1.3.6.1.4.1.5962.99.1.172459500.862792875.1640849999340.2.0'
    slide = ds.get_slide(study_instance_uid, series_instance_uid)
    # Load imaging into in memory cache.
    cache = slide.init_slide_frame_cache()

    # Pathology Embeddings API Natively expects patches to be 224 x 224 pixels
    width = 224
    height = 224
    # Generate a patch slightly offset from the center on the smallest pyramid level.
    slide_pyramid_level = list(slide.levels)[-1]
    x_coordinate = int((slide_pyramid_level.width - width) / 2) + 100
    y_coordinate = int((slide_pyramid_level.height - height) / 2) + 200
    patch = slide.get_patch(slide_pyramid_level, x_coordinate, y_coordinate, width, height)
    # Defines an endpoint that will generate embeddings using the Google Research Endpoint.
    # See linear classifier documentation for examples that illustrate how to
    # generate embeddings using the model externalized on Hugging Face.
    endpoint = patch_embedding_endpoints.V2PatchEmbeddingEndpoint()

    # Generates embedding for a single patch.
    # See EZ-WSI embedding documentation for examples which illustrate how
    # to generate embeddings in batch for multiple patches or entire slides.
    embedding = patch_embedding.get_patch_embedding(endpoint, patch)
    # Display embedding
    print(embedding)

    # optionally retrieve patch pixels to see source imaging for embedding.
    image_bytes = patch.image_bytes()
    # save image bytes, OpenCV expects image bytes to be in BGR ordering.
    cv2.imwrite('test.png', cv2.cvtColor(image_bytes, cv2.COLOR_RGB2BGR))