> [!CAUTION]
> **Caution:** This model is now legacy, but remains available for existing applications. Use [MedSigLIP](https://developers.google.com/health-ai-developer-foundations/medsiglip) for new development.

This document describes the Application Programming Interface (API) for Derm
Foundation when deployed as an HTTPS service endpoint, referred to as *the
service* in this document.

> [!NOTE]
> **Note:** Derm Foundation is available in [other forms](https://developers.google.com/health-ai-developer-foundations/derm-foundation/get-started). This document is only applicable when you deploy Derm Foundation to your own HTTPS endpoint, which is easiest through [Model Garden](https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/derm-foundation).

## Overview

The
[serving source code for Derm Foundation](https://github.com/Google-Health/derm-foundation/tree/master/python/serving/README.md)
can be built and hosted on any API management system, but it's specially
designed to take advantage of
[Vertex AI prediction endpoints](https://cloud.google.com/vertex-ai/docs/predictions/overview).
Therefore, it conforms to Vertex AI's required API signature and implements a
[`predict` method](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.endpoints/predict).

*The service* is designed to support micro batching, not to be mistaken with
[batch jobs](https://developers.google.com/health-ai-developer-foundations/derm-foundation/get-started#launch_a_batch_job).
For every image in the request, if the processing is successful, *the service*
returns a one-dimensional embedding vector in the same order as the image in the
request. Refer to the sections on [API request](https://developers.google.com/health-ai-developer-foundations/derm-foundation/serving-api#request),
[response](https://developers.google.com/health-ai-developer-foundations/derm-foundation/serving-api#response), and [micro batching](https://developers.google.com/health-ai-developer-foundations/derm-foundation/serving-api#micro_batching) for details.

You can provide dermatology images to *the service* either directly within the
request (inlined) or by providing a reference to their location. Inlining the
images in the request is not recommended for large-scale productions; read
[more](https://developers.google.com/health-ai-developer-foundations/derm-foundation/serving-api#inlined_images). When using data storage links *the service* expects
corresponding [OAuth 2.0 bearer tokens](https://www.rfc-editor.org/rfc/rfc6750)
to retrieve the data on your behalf. For detailed information on constructing
API requests and the different ways to provide image data, refer to the
[API request section](https://developers.google.com/health-ai-developer-foundations/derm-foundation/serving-api#request).

> [!TIP]
> **Tip:** When working with large datasets, take advantage of [Google Cloud Storage (GCS)](https://cloud.google.com/storage) and provide links to your data in your service requests instead of inlining the data directly in the request. Explore the benefits and best practices in the [model data documentation](https://developers.google.com/health-ai-developer-foundations/model-data/overview).

To invoke *the service* , consult the [request section](https://developers.google.com/health-ai-developer-foundations/derm-foundation/serving-api#request), compose a
valid request [JSON](https://www.json.org/json-en.html) and send a `POST`
request to your endpoint. If you haven't already deployed Derm Foundation as an
endpoint, the easiest way is through
[Model Garden](https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/derm-foundation).
The following script shows a sample [cURL](https://en.wikipedia.org/wiki/CURL)
command which you can use to invoke *the service* . Set `LOCATION`, `PROJECT_ID`
and `ENDPOINT_ID` to target your endpoint:

> [!TIP]
> **Tip:** Explore service invocations among our [notebooks](https://github.com/Google-Health/derm-foundation/tree/master/notebooks).

    LOCATION="your endpoint location"
    PROJECT_ID="your project ID"
    ENDPOINT_ID="your endpoint ID"
    REQUEST_JSON="path/to/your/request.json"

    curl \
    -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    "https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/endpoints/${ENDPOINT_ID}:predict" \
    -d "@${REQUEST_JSON}"

## Request

An API request can include multiple instances, each conforming to
[this schema](https://github.com/Google-Health/derm-foundation/blob/master/python/serving/vertex_schemata/instance.yaml).
Note that this schema is based on
[Vertex AI PredictSchemata standard](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/PredictSchemata)
and is a partial
[OpenAPI specification](https://en.wikipedia.org/wiki/OpenAPI_Specification).
The complete JSON request has the following structure:

    {
      "instances": [
        {...},
        {...}
      ]
    }

*The service* accepts dermatology images in two ways:

- **Directly within the HTTPS request:** You can include image data as
  [base64-encoded](https://en.wikipedia.org/wiki/Base64) bytes using the
  `input_bytes` JSON field; read more about [inlined images](https://developers.google.com/health-ai-developer-foundations/derm-foundation/serving-api#inlined_images).

- **Indirectly via storage links:** You can provide links to the images stored
  in GCS using the `gcs_uri` JSON field.

To illustrate these methods, the following example JSON request shows
`input_bytes` and `gcs_uri` all in one request. In a real-world scenario, you'll
typically only use one of these options for all images within a single request:

    {
      "instances": [
        {
          "input_bytes": "your base 64 encoded image bytes",
        },
        {
          "gcs_uri": "gs://your-bucket/path/to/image.png",
          "bearer_token": "your-bearer-token",
        }
      ]
    }

### Inlined images

You can inline the images in the [API request](https://developers.google.com/health-ai-developer-foundations/derm-foundation/serving-api#request) as a `base64-encoded`
string in the `input_bytes` JSON field. However, keep in mind most API
management systems enforce a limit on the maximum size of the request payloads.
When Derm Foundation is hosted as a Vertex AI Prediction endpoint,
[Vertex AI quotas](https://cloud.google.com/vertex-ai/docs/quotas) apply.

To optimize the request size, you should compress the images using common image
compression codecs. If you require lossless compression, use
[PNG](https://en.wikipedia.org/wiki/PNG) encoding. If lossy compression is
acceptable, use [JPEG](https://en.wikipedia.org/wiki/JPEG) encoding.

Here is a code snippet for converting compressed JPEG image files from local
file system into a `base64-encoded` string:

    import base64

    def encode_file_bytes(file_path: str) -> str:
      """Reads a file and returns its contents as a base64-encoded string."""

      with open(file_path, 'rb') as imbytes:
        return base64.b64encode(imbytes.read())

Another code snippet for converting uncompressed image bytes into a lossless PNG
format and then converting it into a `base64-encoded` string:

    import base64
    import io
    import numpy as np
    import PIL.Image

    def convert_uncompressed_image_bytes_to_base64(image: np.ndarray) -> str:
      """Converts an uncompressed image array to a base64-encoded PNG string."""

      with io.BytesIO() as compressed_img_bytes:
        with PIL.Image.fromarray(image) as pil_image:
          pil_image.save(compressed_img_bytes, 'png')
        return base64.b64encode(compressed_img_bytes.getvalue())

## Response

An API response can include multiple predictions that correspond to the order of
the instances in the [request](https://developers.google.com/health-ai-developer-foundations/derm-foundation/serving-api#request). Each prediction conforms to
[this schema](https://github.com/Google-Health/derm-foundation/blob/master/python/serving/vertex_schemata/prediction.yaml).
Note that this schema is based on
[Vertex AI PredictSchemata standard](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/PredictSchemata)
and is a partial
[OpenAPI specification](https://en.wikipedia.org/wiki/OpenAPI_Specification).
The complete JSON request has the following structure:

    {
      "predictions": [
        {...},
        {...}
      ],
      "deployedModelId": "model-id",
      "model": "model",
      "modelVersionId": "version-id",
      "modelDisplayName": "model-display-name",
      "metadata": {...}
    }

Each request instance can independently succeed or fail. When succeeded, the
corresponding prediction JSON includes an `embedding` field and when
failed an `error` field. Here is an example of a response to a request with two
instances where the first one has succeeded and the second one failed:

    {
      "predictions": [
        {
          "embedding": [0.1, 0.2, 0.3, 0.4]
        },
        {
          "error": {
            "description": "Some actionable text."
          }
        }
      ],
      "deployedModelId": "model-id",
      "model": "model",
      "modelVersionId": "version-id",
      "modelDisplayName": "model-display-name",
      "metadata": {...}
    }

## Micro batching

> [!TIP]
> **Tip:** Although *the service* is designed to support micro batching, it is not suitable for large batch jobs. Refer to [batch jobs](https://developers.google.com/health-ai-developer-foundations/derm-foundation/get-started#launch_a_batch_job) to learn how you can use Derm Foundation in your batch workflows.

The [API request](https://developers.google.com/health-ai-developer-foundations/derm-foundation/serving-api#request) supports micro batching. You can request embeddings
for multiple images using different instances within the same JSON request:

    {
      "instances": [
        {...},
        {...}
      ]
    }

Keep in mind that the total number of embeddings that you can request in one API
call will be capped by *the service* to a fixed limit. A link to the service
configuration is coming soon.