Performs embedding extraction on images.
The API expects a TFLite model with optional, but strongly recommended, TFLite Model Metadata..
The API supports models with one image input tensor and one or more output tensors. To be more specific, here are the requirements.
- Input image tensor (
kTfLiteUInt8
/kTfLiteFloat32
)- image input of size
[batch x height x width x channels]
. - batch inference is not supported (
batch
is required to be 1). - only RGB inputs are supported (
channels
is required to be 3). - if type is kTfLiteFloat32, NormalizationOptions are required to be attached to the metadata for input normalization.
- image input of size
kTfLiteUInt8
/kTfLiteFloat32
) with shape [1 x N]
where N is the number of dimensions in the produced embeddings.
Nested Classes
class | ImageEmbedder.ImageEmbedderOptions | Options for setting up and ImageEmbedder . |
Public Methods
static double |
cosineSimilarity(Embedding u, Embedding v)
Utility function to compute cosine
similarity between two
Embedding objects. |
static ImageEmbedder |
createFromBuffer(Context context, ByteBuffer modelBuffer)
Creates an
ImageEmbedder instance from a model buffer and default ImageEmbedder.ImageEmbedderOptions . |
static ImageEmbedder |
createFromFile(Context context, String modelPath)
Creates an
ImageEmbedder instance from a model file and default ImageEmbedder.ImageEmbedderOptions . |
static ImageEmbedder |
createFromFile(Context context, File modelFile)
Creates an
ImageEmbedder instance from a model file and default ImageEmbedder.ImageEmbedderOptions . |
static ImageEmbedder |
createFromOptions(Context context, ImageEmbedder.ImageEmbedderOptions options)
Creates an
ImageEmbedder instance from an ImageEmbedder.ImageEmbedderOptions instance. |
ImageEmbedderResult |
embed(MPImage image, ImageProcessingOptions imageProcessingOptions)
Performs embedding extraction on the provided single image.
|
ImageEmbedderResult | |
void |
embedAsync(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)
Sends live image data to perform embedding extraction, and the results will be available via
the
OutputHandler.ResultListener provided in the ImageEmbedder.ImageEmbedderOptions . |
void |
embedAsync(MPImage image, long timestampMs)
Sends live image data to perform embedding extraction with default image processing options,
i.e.
|
ImageEmbedderResult |
embedForVideo(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)
Performs embedding extraction on the provided video frame.
|
ImageEmbedderResult |
embedForVideo(MPImage image, long timestampMs)
Performs embedding extraction on the provided video frame with default image processing
options, i.e.
|
Inherited Methods
Public Methods
public static double cosineSimilarity (Embedding u, Embedding v)
Utility function to compute cosine
similarity between two Embedding
objects.
Parameters
u | |
---|---|
v |
Throws
IllegalArgumentException | if the embeddings are of different types (float vs. quantized), have different sizes, or have an L2-norm of 0. |
---|
public static ImageEmbedder createFromBuffer (Context context, ByteBuffer modelBuffer)
Creates an ImageEmbedder
instance from a model buffer and default ImageEmbedder.ImageEmbedderOptions
.
Parameters
context | an Android ERROR(/Context) . |
---|---|
modelBuffer | a direct ByteBuffer or a ERROR(/MappedByteBuffer) of the embedding
model. |
Throws
if there is an error during ImageEmbedder creation.
|
public static ImageEmbedder createFromFile (Context context, String modelPath)
Creates an ImageEmbedder
instance from a model file and default ImageEmbedder.ImageEmbedderOptions
.
Parameters
context | an Android ERROR(/Context) . |
---|---|
modelPath | path to the embedding model in the assets. |
Throws
if there is an error during ImageEmbedder creation.
|
public static ImageEmbedder createFromFile (Context context, File modelFile)
Creates an ImageEmbedder
instance from a model file and default ImageEmbedder.ImageEmbedderOptions
.
Parameters
context | an Android ERROR(/Context) . |
---|---|
modelFile | the embedding model File instance. |
Throws
IOException | if an I/O error occurs when opening the tflite model file. |
---|---|
if there is an error during ImageEmbedder creation.
|
public static ImageEmbedder createFromOptions (Context context, ImageEmbedder.ImageEmbedderOptions options)
Creates an ImageEmbedder
instance from an ImageEmbedder.ImageEmbedderOptions
instance.
Parameters
context | an Android ERROR(/Context) . |
---|---|
options | an ImageEmbedder.ImageEmbedderOptions instance. |
Throws
if there is an error during ImageEmbedder creation.
|
public ImageEmbedderResult embed (MPImage image, ImageProcessingOptions imageProcessingOptions)
Performs embedding extraction on the provided single image. Only use this method when the
ImageEmbedder
is created with ERROR(/RunningMode.IMAGE)
.
ImageEmbedder
supports the following color space types:
Parameters
image | a MediaPipe MPImage object for processing. |
---|---|
imageProcessingOptions | the ImageProcessingOptions specifying how to process the
input image before running inference. |
Throws
if there is an internal error. |
public ImageEmbedderResult embed (MPImage image)
Performs embedding extraction on the provided single image with default image processing
options, i.e. using the whole image as region-of-interest and without any rotation applied.
Only use this method when the ImageEmbedder
is created with ERROR(/RunningMode.IMAGE)
.
ImageEmbedder
supports the following color space types:
Parameters
image | a MediaPipe MPImage object for processing. |
---|
Throws
if there is an internal error. |
public void embedAsync (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)
Sends live image data to perform embedding extraction, and the results will be available via
the OutputHandler.ResultListener
provided in the ImageEmbedder.ImageEmbedderOptions
. Only use this method
when the ImageEmbedder
is created with ERROR(/RunningMode.LIVE_STREAM)
.
It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the object detector. The input timestamps must be monotonically increasing.
ImageEmbedder
supports the following color space types:
Parameters
image | a MediaPipe MPImage object for processing. |
---|---|
imageProcessingOptions | the ImageProcessingOptions specifying how to process the
input image before running inference. |
timestampMs | the input timestamp (in milliseconds). |
Throws
if there is an internal error. |
public void embedAsync (MPImage image, long timestampMs)
Sends live image data to perform embedding extraction with default image processing options,
i.e. using the whole image as region-of-interest and without any rotation applied, and the
results will be available via the OutputHandler.ResultListener
provided in the ImageEmbedder.ImageEmbedderOptions
. Only use this method when the ImageEmbedder
is created with
ERROR(/RunningMode.LIVE_STREAM)
.
It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the object detector. The input timestamps must be monotonically increasing.
ImageEmbedder
supports the following color space types:
Parameters
image | a MediaPipe MPImage object for processing. |
---|---|
timestampMs | the input timestamp (in milliseconds). |
Throws
if there is an internal error. |
public ImageEmbedderResult embedForVideo (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)
Performs embedding extraction on the provided video frame. Only use this method when the ImageEmbedder
is created with ERROR(/RunningMode.VIDEO)
.
It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.
ImageEmbedder
supports the following color space types:
Parameters
image | a MediaPipe MPImage object for processing. |
---|---|
imageProcessingOptions | the ImageProcessingOptions specifying how to process the
input image before running inference. |
timestampMs | the input timestamp (in milliseconds). |
Throws
if there is an internal error. |
public ImageEmbedderResult embedForVideo (MPImage image, long timestampMs)
Performs embedding extraction on the provided video frame with default image processing
options, i.e. using the whole image as region-of-interest and without any rotation applied.
Only use this method when the ImageEmbedder
is created with ERROR(/RunningMode.VIDEO)
.
It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.
ImageEmbedder
supports the following color space types:
Parameters
image | a MediaPipe MPImage object for processing. |
---|---|
timestampMs | the input timestamp (in milliseconds). |
Throws
if there is an internal error. |