ObjectDetector

public final class ObjectDetector extends BaseVisionTaskApi


Performs object detection on images.

The API expects a TFLite model with TFLite Model Metadata..

The API supports models with one image input tensor and four output tensors. To be more specific, here are the requirements.

  • Input image tensor (kTfLiteUInt8/kTfLiteFloat32)
    • image input of size [batch x height x width x channels].
    • batch inference is not supported (batch is required to be 1).
    • only RGB inputs are supported (channels is required to be 3).
    • if type is kTfLiteFloat32, NormalizationOptions are required to be attached to the metadata for input normalization.
  • Output tensors must be the 4 outputs of a DetectionPostProcess op, i.e:
    • Location tensor (kTfLiteFloat32):
      • tensor of size [1 x num_results x 4], the inner array representing bounding boxes in the form [top, left, right, bottom].
      • BoundingBoxProperties are required to be attached to the metadata and must specify type=BOUNDARIES and coordinate_type=RATIO.
    • Classes tensor (kTfLiteFloat32):
      • tensor of size [1 x num_results], each value representing the integer index of a class.
      • if label maps are attached to the metadata as TENSOR_VALUE_LABELS associated files, they are used to convert the tensor values into labels.
    • scores tensor (kTfLiteFloat32):
      • tensor of size [1 x num_results], each value representing the score of the detected object.
    • Number of detection tensor (kTfLiteFloat32):
      • integer num_results as a tensor of size [1].

An example of such model can be found on TensorFlow Hub..

Summary

Nested types

Options for setting up an ObjectDetector.

Builder for ObjectDetectorOptions.

Public methods

static ObjectDetector
createFromBuffer(Context context, ByteBuffer modelBuffer)

Creates an ObjectDetector instance from a model buffer and the default ObjectDetectorOptions.

static ObjectDetector
createFromFile(Context context, File modelFile)

Creates an ObjectDetector instance from a model file and the default ObjectDetectorOptions.

static ObjectDetector
createFromFile(Context context, String modelPath)

Creates an ObjectDetector instance from a model file and the default ObjectDetectorOptions.

static ObjectDetector
createFromOptions(
    Context context,
    ObjectDetector.ObjectDetectorOptions detectorOptions
)

Creates an ObjectDetector instance from an ObjectDetectorOptions.

ObjectDetectorResult
detect(MPImage image)

Performs object detection on the provided single image with default image processing options, i.e. without any rotation applied.

ObjectDetectorResult
detect(MPImage image, ImageProcessingOptions imageProcessingOptions)

Performs object detection on the provided single image.

void
detectAsync(MPImage image, long timestampMs)

Sends live image data to perform object detection with default image processing options, i.e. without any rotation applied, and the results will be available via the ResultListener provided in the ObjectDetectorOptions.

void
detectAsync(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

Sends live image data to perform object detection, and the results will be available via the ResultListener provided in the ObjectDetectorOptions.

ObjectDetectorResult
detectForVideo(MPImage image, long timestampMs)

Performs object detection on the provided video frame with default image processing options, i.e. without any rotation applied.

ObjectDetectorResult
detectForVideo(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

Performs object detection on the provided video frame.

Inherited methods

From java.lang.AutoCloseable
abstract void
From com.google.mediapipe.tasks.vision.core.BaseVisionTaskApi
void

Closes and cleans up the MediaPipe vision task.

static RectProto.NormalizedRect
convertToNormalizedRect(
    ImageProcessingOptions imageProcessingOptions,
    MPImage image
)

Converts an ImageProcessingOptions instance into a NormalizedRect protobuf message.

static long
generateResultTimestampMs(RunningMode runningMode, Packet packet)

Generates the timestamp of a vision task result object from the vision task running mode and the output packet.

TaskResult
processImageData(Map<String, Packet> inputPackets)

A synchronous method to process single image inputs.

TaskResult
processImageData(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions
)

A synchronous method to process single image inputs.

TaskResult
processVideoData(Map<String, Packet> inputPackets, long timestampMs)

A synchronous method to process continuous video frames.

TaskResult
processVideoData(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

A synchronous method to process continuous video frames.

void
sendLiveStreamData(Map<String, Packet> inputPackets, long timestampMs)

An asynchronous method to send live stream data to the TaskRunner.

void
sendLiveStreamData(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

An asynchronous method to send live stream data to the TaskRunner.

Public methods

createFromBuffer

public static ObjectDetector createFromBuffer(Context context, ByteBuffer modelBuffer)

Creates an ObjectDetector instance from a model buffer and the default ObjectDetectorOptions.

Parameters
Context context

an Android Context.

ByteBuffer modelBuffer

a direct ByteBuffer or a MappedByteBuffer of the detection model.

Throws
MediaPipeException

if there is an error during ObjectDetector creation.

createFromFile

public static ObjectDetector createFromFile(Context context, File modelFile)

Creates an ObjectDetector instance from a model file and the default ObjectDetectorOptions.

Parameters
Context context

an Android Context.

File modelFile

the detection model File instance.

Throws
java.io.IOException

if an I/O error occurs when opening the tflite model file.

MediaPipeException

if there is an error during ObjectDetector creation.

createFromFile

public static ObjectDetector createFromFile(Context context, String modelPath)

Creates an ObjectDetector instance from a model file and the default ObjectDetectorOptions.

Parameters
Context context

an Android Context.

String modelPath

path to the detection model with metadata in the assets.

Throws
MediaPipeException

if there is an error during ObjectDetector creation.

createFromOptions

public static ObjectDetector createFromOptions(
    Context context,
    ObjectDetector.ObjectDetectorOptions detectorOptions
)

Creates an ObjectDetector instance from an ObjectDetectorOptions.

Parameters
Context context

an Android Context.

ObjectDetector.ObjectDetectorOptions detectorOptions

an ObjectDetectorOptions instance.

Throws
MediaPipeException

if there is an error during ObjectDetector creation.

detect

public ObjectDetectorResult detect(MPImage image)

Performs object detection on the provided single image with default image processing options, i.e. without any rotation applied. Only use this method when the ObjectDetector is created with RunningMode.IMAGE.

ObjectDetector supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.

Throws
MediaPipeException

if there is an internal error.

detect

public ObjectDetectorResult detect(MPImage image, ImageProcessingOptions imageProcessingOptions)

Performs object detection on the provided single image. Only use this method when the ObjectDetector is created with RunningMode.IMAGE.

ObjectDetector supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.

ImageProcessingOptions imageProcessingOptions

the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying regionOfInterest will result in this method throwing an IllegalArgumentException.

Throws
java.lang.IllegalArgumentException

if the ImageProcessingOptions specify a region-of-interest.

MediaPipeException

if there is an internal error.

detectAsync

public void detectAsync(MPImage image, long timestampMs)

Sends live image data to perform object detection with default image processing options, i.e. without any rotation applied, and the results will be available via the ResultListener provided in the ObjectDetectorOptions. Only use this method when the ObjectDetector is created with RunningMode.LIVE_STREAM.

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the object detector. The input timestamps must be monotonically increasing.

ObjectDetector supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.

long timestampMs

the input timestamp (in milliseconds).

Throws
MediaPipeException

if there is an internal error.

detectAsync

public void detectAsync(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

Sends live image data to perform object detection, and the results will be available via the ResultListener provided in the ObjectDetectorOptions. Only use this method when the ObjectDetector is created with RunningMode.LIVE_STREAM.

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the object detector. The input timestamps must be monotonically increasing.

ObjectDetector supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.

ImageProcessingOptions imageProcessingOptions

the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying regionOfInterest will result in this method throwing an IllegalArgumentException.

long timestampMs

the input timestamp (in milliseconds).

Throws
java.lang.IllegalArgumentException

if the ImageProcessingOptions specify a region-of-interest.

MediaPipeException

if there is an internal error.

detectForVideo

public ObjectDetectorResult detectForVideo(MPImage image, long timestampMs)

Performs object detection on the provided video frame with default image processing options, i.e. without any rotation applied. Only use this method when the ObjectDetector is created with RunningMode.VIDEO.

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ObjectDetector supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.

long timestampMs

the input timestamp (in milliseconds).

Throws
MediaPipeException

if there is an internal error.

detectForVideo

public ObjectDetectorResult detectForVideo(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

Performs object detection on the provided video frame. Only use this method when the ObjectDetector is created with RunningMode.VIDEO.

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ObjectDetector supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.

ImageProcessingOptions imageProcessingOptions

the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying regionOfInterest will result in this method throwing an IllegalArgumentException.

long timestampMs

the input timestamp (in milliseconds).

Throws
java.lang.IllegalArgumentException

if the ImageProcessingOptions specify a region-of-interest.

MediaPipeException

if there is an internal error.