Attention: This MediaPipe Solutions Preview is an early release. Learn more.

Image embedding guide for Web

Stay organized with collections Save and categorize content based on your preferences.

The MediaPipe Image Embedder task lets you embed images into high-dimensional feature vectors representing their meaning. These instructions show you how to use the Image Embedder for Node and web apps.

For more information about the capabilities, models, and configuration options of this task, see the Overview.

Setup

This section describes key steps for setting up your development environment and code projects specifically to use Image Embedder. For general information on setting up your development environment for using MediaPipe tasks, including platform version requirements, see the Setup guide for Web.

JavaScript packages

Image Embedder code is available through the MediaPipe @mediapipe/tasks-vision NPM package. You can find and download these libraries from links provided in the platform Setup guide.

You can install the required packages with the following code for local staging using the following command:

npm install @mediapipe/tasks-vision

If you want to import the task code via a content delivery network (CDN) service, add the following code in the tag in your HTML file:

<!-- Replace "my-cdn-service.com" with your CDN -->
<head>
  <script src="https://my-cdn-service.com/npm/@mediapipe/tasks-vision/vision_bundle.js"
    crossorigin="anonymous"></script>
</head>

Model

The MediaPipe Image Embedder task requires a trained model that is compatible with this task. For more information on available trained models for Image Embedder, see the task overview Models section.

Select and download a model, and then store it within your project directory:

<dev-project-root>/app/shared/models/

Create the task

Specify a model path

You can create a task with the default options using the createFromModelPath() method:

const vision = await FilesetResolver.forVisionTasks("path/to/wasm/root");
const imageEmbedder = ImageEmbedder.createFromModelPath(
    vision,
   "model.tflite"
);

Specify the model buffer

If your model is already loaded into memory, you can use the createFromModelBuffer() method:

const vision = await FilesetResolver.forVisionTasks("path/to/wasm/root");
const imageEmbedder = ImageEmbedder.createFromModelBuffer(
    vision, 
    new Uint8Array(...)
);

Specify custom options

The MediaPipe Image Embedder task uses the createFromOptions function to set up the task. The createFromOptions function accepts values for configuration options. For more information on configuration options, see Configuration options.

The following code demonstrates how to build and configure the task with custom options:

const vision = await FilesetResolver.forVisionTasks("path/to/wasm/root");
const imageEmbedder = await ImageEmbedder.createFromOptions(
    vision,
    {
      baseOptions: {
        modelAssetPath: "model.tflite"
      },
      quantize: true
    });

Configuration options

This task has the following configuration options for Web applications:

Option Name Description Value Range Default Value
l2Normalize Whether to normalize the returned feature vector with L2 norm. Use this option only if the model does not already contain a native L2_NORMALIZATION TFLite Op. In most cases, this is already the case and L2 normalization is thus achieved through TFLite inference with no need for this option. Boolean False
quantize Whether the returned embedding should be quantized to bytes via scalar quantization. Embeddings are implicitly assumed to be unit-norm and therefore any dimension is guaranteed to have a value in [-1.0, 1.0]. Use the l2Normalize option if this is not the case. Boolean False

Prepare data

Image Embedder can embed images in any format supported by the host browser. The task also handles data input preprocessing, including resizing, rotation and value normalization.

Calls to the Image Embedder embed() and embedFromVideo() methods run synchronously and block the user interface thread. If you want to extract feature vectors from video frames, each embedding will block the main thread. You can prevent this by implementing web workers to run the embed() and embedFromVideo() methods on another thread.

Run the task

The Image Embedder uses the embed() (with running mode image) and embedFromVideo() (with running mode video) methods to trigger inferences. The Image Embedder API will return the embedding vectors for the input image.

The following code demonstrates how execute the processing with the task model:

Image

const image = document.getElementById("image") as HTMLImageElement;
const imageEmbedderResult = imageEmbedder.embed(image);

Video

await imageEmbedder.setOptions({ runningMode: "video" });

let lastVideoTime = -1;
function renderLoop(): void {
  const video = document.getElementById("video");

  if (video.currentTime !== lastVideoTime) {
    const imageEmbedderResult = imageEmbedder.embedFromVideo(video);
    processResult(imageEmbedderResult);
    lastVideoTime = video.currentTime;
  }

  requestAnimationFrame(() => {
    renderLoop();
  });
}

Handle and display results

Upon running inference, the Image Embedder task returns an ImageEmbedderResult object which contains the embedding vectors for the input image or frame.

The following shows an example of the output data from this task:

ImageEmbedderResult:
  Embedding #0 (sole embedding head):
    float_embedding: {0.0, 0.0, ..., 0.0, 1.0, 0.0, 0.0, 2.0}
    head_index: 0

This result was obtained by embedding the following image:

You can compare the semantic similarity of two embeddings using the ImageEmbedder.cosineSimilarity function. See the following code for an example.

// Compute cosine similarity.
const similarity = ImageEmbedder.cosineSimilarity(
  imageEmbedderResult.embeddings[0],
  otherEmbedderResult.embeddings[0]);