ImageSegmenter

public final class ImageSegmenter

Performs image segmentation on images.

Note that, in addition to the standard segmentation API, ERROR(/segment) and ERROR(/segmentForVideo), that take an input image and return the outputs, but involves deep copy of the returns, ImageSegmenter also supports the callback API, ERROR(/segmentWithResultListener) and ERROR(/segmentForVideoWithResultListener), which allow you to access the outputs through zero copy.

The callback API is available for all RunningMode in ImageSegmenter. Set OutputHandler.ResultListener in ImageSegmenter.ImageSegmenterOptions properly to use the callback API.

The API expects a TFLite model with,TFLite Model Metadata..

• Input image MPImage
  • The image that image segmenter runs on.
• Output ImageSegmenterResult ImageSegmenterResult
  • An ImageSegmenterResult containing segmented masks.

Nested Classes

class	ImageSegmenter.ImageSegmenterOptions		Options for setting up an ImageSegmenter.
class	ImageSegmenter.SegmentationOptions		Options for configuring runtime behavior of ImageSegmenter.

Public Methods

static ImageSegmenter	createFromOptions(Context context, ImageSegmenter.ImageSegmenterOptions segmenterOptions) Creates an ImageSegmenter instance from an ImageSegmenter.ImageSegmenterOptions.
List<String>	getLabels() Get the category label list of the ImageSegmenter can recognize.
ImageSegmenterResult	segment(MPImage image, ImageSegmenter.SegmentationOptions segmentationOptions) Performs image segmentation on the provided single image.
ImageSegmenterResult	segment(MPImage image, ImageProcessingOptions imageProcessingOptions) Performs image segmentation on the provided single image.
ImageSegmenterResult	segment(MPImage image) Performs image segmentation on the provided single image with default image processing options, i.e.
void	segmentAsync(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs) Sends live image data to perform image segmentation, and the results will be available via the OutputHandler.ResultListener provided in the ImageSegmenter.ImageSegmenterOptions.
void	segmentAsync(MPImage image, ImageSegmenter.SegmentationOptions segmentationOptions, long timestampMs) Sends live image data to perform image segmentation, and the results will be available via the OutputHandler.ResultListener provided in the ImageSegmenter.ImageSegmenterOptions.
void	segmentAsync(MPImage image, long timestampMs) Sends live image data to perform image segmentation with default image processing options, i.e.
ImageSegmenterResult	segmentForVideo(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs) Performs image segmentation on the provided video frame.
ImageSegmenterResult	segmentForVideo(MPImage image, long timestampMs) Performs image segmentation on the provided video frame with default image processing options, i.e.
ImageSegmenterResult	segmentForVideo(MPImage image, ImageSegmenter.SegmentationOptions segmentationOptions, long timestampMs) Performs image segmentation on the provided video frame.
void	segmentForVideoWithResultListener(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs) Performs image segmentation on the provided video frame, and provides zero-copied results via OutputHandler.ResultListener in ImageSegmenter.ImageSegmenterOptions.
void	segmentForVideoWithResultListener(MPImage image, ImageSegmenter.SegmentationOptions segmentationOptions, long timestampMs) Performs image segmentation on the provided video frame, and provides zero-copied results via OutputHandler.ResultListener in ImageSegmenter.ImageSegmenterOptions.
void	segmentForVideoWithResultListener(MPImage image, long timestampMs) Performs image segmentation on the provided video frame with default image processing options, i.e.
void	segmentWithResultListener(MPImage image, ImageProcessingOptions imageProcessingOptions) Performs image segmentation on the provided single image, and provides zero-copied results via OutputHandler.ResultListener in ImageSegmenter.ImageSegmenterOptions.
void	segmentWithResultListener(MPImage image) Performs image segmentation on the provided single image with default image processing options, i.e.
void	segmentWithResultListener(MPImage image, ImageSegmenter.SegmentationOptions segmentationOptions) Performs image segmentation on the provided single image, and provides zero-copied results via OutputHandler.ResultListener in ImageSegmenter.ImageSegmenterOptions.

Inherited Methods

From class com.google.mediapipe.tasks.vision.core.BaseVisionTaskApi

void	close() Closes and cleans up the MediaPipe vision task.
static long	generateResultTimestampMs(RunningMode runningMode, Packet packet) Generates the timestamp of a vision task result object from the vision task running mode and the output packet.

From class java.lang.Object

boolean	equals(Object arg0)
final Class<?>	getClass()
int	hashCode()
final void	notify()
final void	notifyAll()
String	toString()
final void	wait(long arg0, int arg1)
final void	wait(long arg0)
final void	wait()

From interface java.lang.AutoCloseable

abstract void

close()

Public Methods

public static ImageSegmenter createFromOptions (Context context, ImageSegmenter.ImageSegmenterOptions segmenterOptions)

Creates an ImageSegmenter instance from an ImageSegmenter.ImageSegmenterOptions.

Parameters

context	an Android ERROR(/Context).
segmenterOptions	an ImageSegmenter.ImageSegmenterOptions instance.

Throws

	if there is an error during ImageSegmenter creation.

public List<String> getLabels ()

Get the category label list of the ImageSegmenter can recognize. For CATEGORY_MASK type, the index in the category mask corresponds to the category in the label list. For CONFIDENCE_MASK type, the output mask list at index corresponds to the category in the label list.

If there is no labelmap provided in the model file, empty label list is returned.

public ImageSegmenterResult segment (MPImage image, ImageSegmenter.SegmentationOptions segmentationOptions)

Performs image segmentation on the provided single image. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.IMAGE). TODO update java doc for input image format.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
segmentationOptions	the ImageSegmenter.SegmentationOptions used to configure the runtime behavior of the ImageSegmenter.

Throws

	if there is an internal error. Or if ImageSegmenter is created with a OutputHandler.ResultListener.

public ImageSegmenterResult segment (MPImage image, ImageProcessingOptions imageProcessingOptions)

Performs image segmentation on the provided single image. The output mask has the same size as the input image. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.IMAGE). TODO update java doc for input image format.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
imageProcessingOptions	the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying ImageProcessingOptions.regionOfInterest() will result in this method throwing an IllegalArgumentException.

Throws

IllegalArgumentException	if the ImageProcessingOptions specify a region-of-interest.
	if there is an internal error. Or if ImageSegmenter is created with a OutputHandler.ResultListener.

public ImageSegmenterResult segment (MPImage image)

Performs image segmentation on the provided single image with default image processing options, i.e. without any rotation applied. The output mask has the same size as the input image. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.IMAGE). TODO update java doc for input image format.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.

Throws

	if there is an internal error. Or if ImageSegmenter is created with a OutputHandler.ResultListener.

public void segmentAsync (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Sends live image data to perform image segmentation, and the results will be available via the OutputHandler.ResultListener provided in the ImageSegmenter.ImageSegmenterOptions. The output mask has the same size as the input image. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the image segmenter. The input timestamps must be monotonically increasing.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
imageProcessingOptions	the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying ImageProcessingOptions.regionOfInterest() will result in this method throwing an IllegalArgumentException.
timestampMs	the input timestamp (in milliseconds).

Throws

IllegalArgumentException	if the ImageProcessingOptions specify a region-of-interest.
	if there is an internal error.

public void segmentAsync (MPImage image, ImageSegmenter.SegmentationOptions segmentationOptions, long timestampMs)

Sends live image data to perform image segmentation, and the results will be available via the OutputHandler.ResultListener provided in the ImageSegmenter.ImageSegmenterOptions. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the image segmenter. The input timestamps must be monotonically increasing.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
segmentationOptions	the ImageSegmenter.SegmentationOptions used to configure the runtime behavior of the ImageSegmenter.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public void segmentAsync (MPImage image, long timestampMs)

Sends live image data to perform image segmentation with default image processing options, i.e. without any rotation applied, and the results will be available via the OutputHandler.ResultListener provided in the ImageSegmenter.ImageSegmenterOptions. The output mask has the same size as the input image. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the image segmenter. The input timestamps must be monotonically increasing.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public ImageSegmenterResult segmentForVideo (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Performs image segmentation on the provided video frame. The output mask has the same size as the input image. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
imageProcessingOptions	the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying ImageProcessingOptions.regionOfInterest() will result in this method throwing an IllegalArgumentException.
timestampMs	the input timestamp (in milliseconds).

Throws

IllegalArgumentException	if the ImageProcessingOptions specify a region-of-interest.
	if there is an internal error. Or if ImageSegmenter is created with a OutputHandler.ResultListener.

public ImageSegmenterResult segmentForVideo (MPImage image, long timestampMs)

Performs image segmentation on the provided video frame with default image processing options, i.e. without any rotation applied. The output mask has the same size as the input image. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error. Or if ImageSegmenter is created with a OutputHandler.ResultListener.

public ImageSegmenterResult segmentForVideo (MPImage image, ImageSegmenter.SegmentationOptions segmentationOptions, long timestampMs)

Performs image segmentation on the provided video frame. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
segmentationOptions	the ImageSegmenter.SegmentationOptions used to configure the runtime behavior of the ImageSegmenter.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error. Or if ImageSegmenter is created with a OutputHandler.ResultListener.

public void segmentForVideoWithResultListener (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Performs image segmentation on the provided video frame, and provides zero-copied results via OutputHandler.ResultListener in ImageSegmenter.ImageSegmenterOptions. The output mask has the same size as the input image. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
imageProcessingOptions
timestampMs	the input timestamp (in milliseconds).

Throws

IllegalArgumentException	if the ImageProcessingOptions specify a region-of-interest.
	if there is an internal error. Or if ImageSegmenter is not created with OutputHandler.ResultListener set in ImageSegmenter.ImageSegmenterOptions.

public void segmentForVideoWithResultListener (MPImage image, ImageSegmenter.SegmentationOptions segmentationOptions, long timestampMs)

Performs image segmentation on the provided video frame, and provides zero-copied results via OutputHandler.ResultListener in ImageSegmenter.ImageSegmenterOptions. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
segmentationOptions	the ImageSegmenter.SegmentationOptions used to configure the runtime behavior of the ImageSegmenter.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error. Or if ImageSegmenter is not created with OutputHandler.ResultListener set in ImageSegmenter.ImageSegmenterOptions.

public void segmentForVideoWithResultListener (MPImage image, long timestampMs)

Performs image segmentation on the provided video frame with default image processing options, i.e. without any rotation applied, and provides zero-copied results via OutputHandler.ResultListener in ImageSegmenter.ImageSegmenterOptions. The output mask has the same size as the input image. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error. Or if ImageSegmenter is not created with OutputHandler.ResultListener set in ImageSegmenter.ImageSegmenterOptions.

public void segmentWithResultListener (MPImage image, ImageProcessingOptions imageProcessingOptions)

Performs image segmentation on the provided single image, and provides zero-copied results via OutputHandler.ResultListener in ImageSegmenter.ImageSegmenterOptions. The output mask has the same size as the input image. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.IMAGE).

TODO update java doc for input image format.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
imageProcessingOptions	the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying ImageProcessingOptions.regionOfInterest() will result in this method throwing an IllegalArgumentException.

Throws

IllegalArgumentException	if the ImageProcessingOptions specify a region-of-interest.
	if there is an internal error. Or if ImageSegmenter is not created with OutputHandler.ResultListener set in ImageSegmenter.ImageSegmenterOptions.

public void segmentWithResultListener (MPImage image)

Performs image segmentation on the provided single image with default image processing options, i.e. without any rotation applied, and provides zero-copied results via OutputHandler.ResultListener in ImageSegmenter.ImageSegmenterOptions. The output mask has the same size as the input image. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.IMAGE).

TODO update java doc for input image format.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.

Throws

	if there is an internal error. Or if ImageSegmenter is not created with OutputHandler.ResultListener set in ImageSegmenter.ImageSegmenterOptions.

public void segmentWithResultListener (MPImage image, ImageSegmenter.SegmentationOptions segmentationOptions)

Performs image segmentation on the provided single image, and provides zero-copied results via OutputHandler.ResultListener in ImageSegmenter.ImageSegmenterOptions. Only use this method when the ImageSegmenter is created with ERROR(/RunningMode.IMAGE).

TODO update java doc for input image format.

ImageSegmenter supports the following color space types:

• ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe MPImage object for processing.
segmentationOptions	the ImageSegmenter.SegmentationOptions used to configure the runtime behavior of the ImageSegmenter.

Throws

	if there is an internal error. Or if ImageSegmenter is not created with OutputHandler.ResultListener set in ImageSegmenter.ImageSegmenterOptions.