Attention: This MediaPipe Solutions Preview is an early release. Learn more

ImageClassifier

public final class ImageClassifier

Performs classification on images.

The API expects a TFLite model with optional, but strongly recommended, TFLite Model Metadata..

The API supports models with one image input tensor and one or more output tensors. To be more specific, here are the requirements.

Input image tensor (kTfLiteUInt8/kTfLiteFloat32)
- image input of size [batch x height x width x channels].
- batch inference is not supported (batch is required to be 1).
- only RGB inputs are supported (channels is required to be 3).
- if type is kTfLiteFloat32, NormalizationOptions are required to be attached to the metadata for input normalization.
At least one output tensor (kTfLiteUInt8/kTfLiteFloat32) with:
- N classes and either 2 or 4 dimensions, i.e. [1 x N] or [1 x 1 x 1 x N]
- optional (but recommended) label map(s) as AssociatedFile-s with type TENSOR_AXIS_LABELS, containing one label per line. The first such AssociatedFile (if any) is used to fill the class_name field of the results. The display_name field is filled from the AssociatedFile (if any) whose locale matches the display_names_locale field of the ImageClassifierOptions used at creation time ("en" by default, i.e. English). If none of these are available, only the index field of the results will be filled.
- optional score calibration can be attached using ScoreCalibrationOptions and an AssociatedFile with type TENSOR_AXIS_SCORE_CALIBRATION. See metadata_schema.fbs for more details.

An example of such model can be found TensorFlow Hub.

Nested Classes

class ImageClassifier.ImageClassifierOptions Options for setting up and ImageClassifier.

Public Methods

ImageClassifierResult	classify(MPImage image) Performs classification on the provided single image with default image processing options, i.e.
ImageClassifierResult	classify(MPImage image, ImageProcessingOptions imageProcessingOptions) Performs classification on the provided single image.
void	classifyAsync(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs) Sends live image data to perform classification, and the results will be available via the `OutputHandler.ResultListener` provided in the `ImageClassifier.ImageClassifierOptions`.
void	classifyAsync(MPImage image, long timestampMs) Sends live image data to perform classification with default image processing options, i.e.
ImageClassifierResult	classifyForVideo(MPImage image, long timestampMs) Performs classification on the provided video frame with default image processing options, i.e.
ImageClassifierResult	classifyForVideo(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs) Performs classification on the provided video frame.
static ImageClassifier	createFromBuffer(Context context, ByteBuffer modelBuffer) Creates an `ImageClassifier` instance from a model buffer and default `ImageClassifier.ImageClassifierOptions`.
static ImageClassifier	createFromFile(Context context, String modelPath) Creates an `ImageClassifier` instance from a model file and default `ImageClassifier.ImageClassifierOptions`.
static ImageClassifier	createFromFile(Context context, File modelFile) Creates an `ImageClassifier` instance from a model file and default `ImageClassifier.ImageClassifierOptions`.
static ImageClassifier	createFromOptions(Context context, ImageClassifier.ImageClassifierOptions options) Creates an `ImageClassifier` instance from an `ImageClassifier.ImageClassifierOptions` instance.

Inherited Methods

From class com.google.mediapipe.tasks.vision.core.BaseVisionTaskApi

void	close() Closes and cleans up the MediaPipe vision task.
static long	generateResultTimestampMs(RunningMode runningMode, Packet packet) Generates the timestamp of a vision task result object from the vision task running mode and the output packet.

From class java.lang.Object

boolean	equals(Object arg0)
final Class<?>	getClass()
int	hashCode()
final void	notify()
final void	notifyAll()
String	toString()
final void	wait(long arg0, int arg1)
final void	wait(long arg0)
final void	wait()

From interface java.lang.AutoCloseable

abstract void

close()

Public Methods

public ImageClassifierResult classify (MPImage image)

Performs classification on the provided single image with default image processing options, i.e. using the whole image as region-of-interest and without any rotation applied. Only use this method when the ImageClassifier is created with ERROR(/RunningMode.IMAGE).

ImageClassifier supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.

Throws

	if there is an internal error.

public ImageClassifierResult classify (MPImage image, ImageProcessingOptions imageProcessingOptions)

Performs classification on the provided single image. Only use this method when the ImageClassifier is created with ERROR(/RunningMode.IMAGE).

ImageClassifier supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference.

Throws

	if there is an internal error.

public void classifyAsync (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Sends live image data to perform classification, and the results will be available via the OutputHandler.ResultListener provided in the ImageClassifier.ImageClassifierOptions. Only use this method when the ImageClassifier is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the object detector. The input timestamps must be monotonically increasing.

ImageClassifier supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public void classifyAsync (MPImage image, long timestampMs)

Sends live image data to perform classification with default image processing options, i.e. using the whole image as region-of-interest and without any rotation applied, and the results will be available via the OutputHandler.ResultListener provided in the ImageClassifier.ImageClassifierOptions. Only use this method when the ImageClassifier is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the object detector. The input timestamps must be monotonically increasing.

ImageClassifier supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public ImageClassifierResult classifyForVideo (MPImage image, long timestampMs)

Performs classification on the provided video frame with default image processing options, i.e. using the whole image as region-of-interest and without any rotation applied. Only use this method when the ImageClassifier is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ImageClassifier supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public ImageClassifierResult classifyForVideo (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Performs classification on the provided video frame. Only use this method when the ImageClassifier is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ImageClassifier supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public static ImageClassifier createFromBuffer (Context context, ByteBuffer modelBuffer)

Creates an ImageClassifier instance from a model buffer and default ImageClassifier.ImageClassifierOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelBuffer	a direct `ByteBuffer` or a `ERROR(/MappedByteBuffer)` of the classification model.

Throws

	if there is an error during `ImageClassifier` creation.

public static ImageClassifier createFromFile (Context context, String modelPath)

Creates an ImageClassifier instance from a model file and default ImageClassifier.ImageClassifierOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelPath	path to the classification model in the assets.

Throws

	if there is an error during `ImageClassifier` creation.

public static ImageClassifier createFromFile (Context context, File modelFile)

Creates an ImageClassifier instance from a model file and default ImageClassifier.ImageClassifierOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelFile	the classification model `File` instance.

Throws

IOException	if an I/O error occurs when opening the tflite model file.
	if there is an error during `ImageClassifier` creation.

public static ImageClassifier createFromOptions (Context context, ImageClassifier.ImageClassifierOptions options)

Creates an ImageClassifier instance from an ImageClassifier.ImageClassifierOptions instance.

Parameters

context	an Android `ERROR(/Context)`.
options	an `ImageClassifier.ImageClassifierOptions` instance.

Throws

	if there is an error during `ImageClassifier` creation.