Attention: This MediaPipe Solutions Preview is an early release. Learn more.

GestureRecognizer

Stay organized with collections Save and categorize content based on your preferences.
public final class GestureRecognizer

Performs gesture recognition on images.

This API expects a pre-trained hand gesture model asset bundle, or a custom one created using Model Maker. See .

  • Input image ERROR(/MPImage)
    • The image that gesture recognition runs on.
  • Output GestureRecognizerResult GestureRecognizerResult
    • A GestureRecognizerResult containing hand landmarks and recognized hand gestures.

Nested Classes

class GestureRecognizer.GestureRecognizerOptions Options for setting up an GestureRecognizer

Public Methods

static GestureRecognizer
createFromBuffer(Context context, ByteBuffer modelBuffer)
Creates a GestureRecognizer instance from a model buffer and the default GestureRecognizer.GestureRecognizerOptions.
static GestureRecognizer
createFromFile(Context context, String modelPath)
Creates a GestureRecognizer instance from a model file and the default GestureRecognizer.GestureRecognizerOptions.
static GestureRecognizer
createFromFile(Context context, File modelFile)
Creates a GestureRecognizer instance from a model file and the default GestureRecognizer.GestureRecognizerOptions.
static GestureRecognizer
GestureRecognizerResult
recognize(MPImage image, ImageProcessingOptions imageProcessingOptions)
Performs gesture recognition on the provided single image.
GestureRecognizerResult
recognize(MPImage image)
Performs gesture recognition on the provided single image with default image processing options, i.e.
void
recognizeAsync(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)
Sends live image data to perform gesture recognition, and the results will be available via the OutputHandler.ResultListener provided in the GestureRecognizer.GestureRecognizerOptions.
void
recognizeAsync(MPImage image, long timestampMs)
Sends live image data to perform gesture recognition with default image processing options, i.e.
GestureRecognizerResult
recognizeForVideo(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)
Performs gesture recognition on the provided video frame.
GestureRecognizerResult
recognizeForVideo(MPImage image, long timestampMs)
Performs gesture recognition on the provided video frame with default image processing options, i.e.

Inherited Methods

Public Methods

public static GestureRecognizer createFromBuffer (Context context, ByteBuffer modelBuffer)

Creates a GestureRecognizer instance from a model buffer and the default GestureRecognizer.GestureRecognizerOptions.

Parameters
context an Android ERROR(/Context).
modelBuffer a direct ByteBuffer or a ERROR(/MappedByteBuffer) of the detection model.
Throws
if there is an error during GestureRecognizer creation.

public static GestureRecognizer createFromFile (Context context, String modelPath)

Creates a GestureRecognizer instance from a model file and the default GestureRecognizer.GestureRecognizerOptions.

Parameters
context an Android ERROR(/Context).
modelPath path to the gesture recognition model with metadata in the assets.
Throws
if there is an error during GestureRecognizer creation.

public static GestureRecognizer createFromFile (Context context, File modelFile)

Creates a GestureRecognizer instance from a model file and the default GestureRecognizer.GestureRecognizerOptions.

Parameters
context an Android ERROR(/Context).
modelFile the gesture recognition model File instance.
Throws
IOException if an I/O error occurs when opening the tflite model file.
if there is an error during GestureRecognizer creation.

public static GestureRecognizer createFromOptions (Context context, GestureRecognizer.GestureRecognizerOptions recognizerOptions)

Parameters
context an Android ERROR(/Context).
recognizerOptions a GestureRecognizer.GestureRecognizerOptions instance.
Throws
if there is an error during GestureRecognizer creation.

public GestureRecognizerResult recognize (MPImage image, ImageProcessingOptions imageProcessingOptions)

Performs gesture recognition on the provided single image. Only use this method when the GestureRecognizer is created with ERROR(/RunningMode.IMAGE). TODO update java doc for input image format.

GestureRecognizer supports the following color space types:

Parameters
image a MediaPipe ERROR(/MPImage) object for processing.
imageProcessingOptions the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying ImageProcessingOptions.regionOfInterest() will result in this method throwing an IllegalArgumentException.
Throws
IllegalArgumentException if the ImageProcessingOptions specify a region-of-interest.
if there is an internal error.

public GestureRecognizerResult recognize (MPImage image)

Performs gesture recognition on the provided single image with default image processing options, i.e. without any rotation applied. Only use this method when the GestureRecognizer is created with ERROR(/RunningMode.IMAGE). TODO update java doc for input image format.

GestureRecognizer supports the following color space types:

Parameters
image a MediaPipe ERROR(/MPImage) object for processing.
Throws
if there is an internal error.

public void recognizeAsync (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Sends live image data to perform gesture recognition, and the results will be available via the OutputHandler.ResultListener provided in the GestureRecognizer.GestureRecognizerOptions. Only use this method when the ERROR(/GestureRecognition) is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the gesture recognizer. The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

Parameters
image a MediaPipe ERROR(/MPImage) object for processing.
imageProcessingOptions the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying ImageProcessingOptions.regionOfInterest() will result in this method throwing an IllegalArgumentException.
timestampMs the input timestamp (in milliseconds).
Throws
IllegalArgumentException if the ImageProcessingOptions specify a region-of-interest.
if there is an internal error.

public void recognizeAsync (MPImage image, long timestampMs)

Sends live image data to perform gesture recognition with default image processing options, i.e. without any rotation applied, and the results will be available via the OutputHandler.ResultListener provided in the GestureRecognizer.GestureRecognizerOptions. Only use this method when the ERROR(/GestureRecognition) is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the gesture recognizer. The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

Parameters
image a MediaPipe ERROR(/MPImage) object for processing.
timestampMs the input timestamp (in milliseconds).
Throws
if there is an internal error.

public GestureRecognizerResult recognizeForVideo (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Performs gesture recognition on the provided video frame. Only use this method when the GestureRecognizer is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

Parameters
image a MediaPipe ERROR(/MPImage) object for processing.
imageProcessingOptions the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying ImageProcessingOptions.regionOfInterest() will result in this method throwing an IllegalArgumentException.
timestampMs the input timestamp (in milliseconds).
Throws
IllegalArgumentException if the ImageProcessingOptions specify a region-of-interest.
if there is an internal error.

public GestureRecognizerResult recognizeForVideo (MPImage image, long timestampMs)

Performs gesture recognition on the provided video frame with default image processing options, i.e. without any rotation applied. Only use this method when the GestureRecognizer is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

Parameters
image a MediaPipe ERROR(/MPImage) object for processing.
timestampMs the input timestamp (in milliseconds).
Throws
if there is an internal error.