# ImageSegmenter class ImageSegmenter: NSObject Class that performs segmentation on images. The API expects a TFLite model with mandatory TFLite Model Metadata. - ` ### [init(modelPath:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageSegmenter.md.txt#/c:objc(cs)MPPImageSegmenter(im)initWithModelPath:error:) ` Creates a new instance of `ImageSegmenter` from an absolute path to a TensorFlow Lite model file stored locally on the device and the default `ImageSegmenterOptions`. #### Declaration Swift convenience init(modelPath: String) throws #### Parameters |---|---| | `modelPath` | An absolute path to a TensorFlow Lite model file stored locally on the device. | #### Return Value A new instance of `ImageSegmenter` with the given model path. `nil` if there is an error in initializing the image segmenter. - ` ### [init(options:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageSegmenter.md.txt#/c:objc(cs)MPPImageSegmenter(im)initWithOptions:error:) ` Creates a new instance of `ImageSegmenter` from the given `ImageSegmenterOptions`. #### Declaration Swift init(options: ImageSegmenterOptions) throws #### Parameters |---|---| | `options` | The options of type `ImageSegmenterOptions` to use for configuring the `ImageSegmenter`. | #### Return Value A new instance of `ImageSegmenter` with the given options. `nil` if there is an error in initializing the image segmenter. - ` ### [labels](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageSegmenter.md.txt#/c:objc(cs)MPPImageSegmenter(py)labels) ` Get the category label list of the `ImageSegmenter` can recognize. For CATEGORY_MASK type, the index in the category mask corresponds to the category in the label list. For CONFIDENCE_MASK type, the output mask list at index corresponds to the category in the label list. If there is no labelmap provided in the model file, empty array is returned. #### Declaration Swift var labels: [String] { get } - ` ### [segment(image:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageSegmenter.md.txt#/c:objc(cs)MPPImageSegmenter(im)segmentImage:withCompletionHandler:-async) ` Performs segmentation on the provided MPPImage using the whole image as region of interest and invokes the given completion handler block with the response. The method returns synchronously once the completion handler returns. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageSegmenter` is created with running mode, `image`. This method supports segmentation of RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format. If your `MPImage` has a source type of `image` ensure that the color space is RGB with an Alpha channel. #### Declaration Swift func segment(image: MPImage) async throws -> ImageSegmenterResult #### Parameters |---|---| | `image` | The `MPImage` on which segmentation is to be performed. | | `completionHandler` | A block to be invoked with the results of performing segmentation on the image. The block takes two arguments, the optional `ImageSegmenterResult` that contains the segmented masks if the segmentation was successful and an optional error populated upon failure. The lifetime of the returned masks is only guaranteed for the duration of the block. | - ` ### [segment(image:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageSegmenter.md.txt#/c:objc(cs)MPPImageSegmenter(im)segmentImage:error:) ` Performs segmentation on the provided MPPImage using the whole image as region of interest. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageSegmenter` is created with running mode, `image`. This method supports segmentation of RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format. If your `MPImage` has a source type of `.image` ensure that the color space is RGB with an Alpha channel. #### Declaration Swift func segment(image: MPImage) throws -> ImageSegmenterResult #### Parameters |---|---| | `image` | The `MPImage` on which segmentation is to be performed. | #### Return Value An `ImageSegmenterResult` that contains the segmented masks. - ` ### [segment(image:completion:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageSegmenter.md.txt#/c:objc(cs)MPPImageSegmenter(im)segmentImage:withCompletionHandler:) ` Performs segmentation on the provided MPPImage using the whole image as region of interest and invokes the given completion handler block with the response. The method returns synchronously once the completion handler returns. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageSegmenter` is created with running mode, `image`. This method supports segmentation of RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format. If your `MPImage` has a source type of `image` ensure that the color space is RGB with an Alpha channel. #### Declaration Swift func segment(image: MPImage, completion completionHandler: @escaping @Sendable (ImageSegmenterResult?, (any Error)?) -> Void) #### Parameters |---|---| | `image` | The `MPImage` on which segmentation is to be performed. | | `completionHandler` | A block to be invoked with the results of performing segmentation on the image. The block takes two arguments, the optional `ImageSegmenterResult` that contains the segmented masks if the segmentation was successful and an optional error populated upon failure. The lifetime of the returned masks is only guaranteed for the duration of the block. | - ` ### [segment(videoFrame:timestampInMilliseconds:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageSegmenter.md.txt#/c:objc(cs)MPPImageSegmenter(im)segmentVideoFrame:timestampInMilliseconds:withCompletionHandler:-async) ` Performs segmentation on the provided video frame of type `MPImage` using the whole image as region of interest invokes the given completion handler block with the response. The method returns synchronously once the completion handler returns. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageSegmenter` is created with running mode, `video`. This method supports segmentation of RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format. If your `MPImage` has a source type of `image` ensure that the color space is RGB with an Alpha channel. #### Declaration Swift func segment(videoFrame image: MPImage, timestampInMilliseconds: Int) async throws -> ImageSegmenterResult #### Parameters |---|---| | `image` | The `MPImage` on which segmentation is to be performed. | | `timestampInMilliseconds` | The video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing. | | `completionHandler` | A block to be invoked with the results of performing segmentation on the image. The block takes two arguments, the optional `ImageSegmenterResult` that contains the segmented masks if the segmentation was successful and an optional error only populated upon failure. The lifetime of the returned masks is only guaranteed for the duration of the block. | - ` ### [segment(videoFrame:timestampInMilliseconds:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageSegmenter.md.txt#/c:objc(cs)MPPImageSegmenter(im)segmentVideoFrame:timestampInMilliseconds:error:) ` Performs segmentation on the provided video frame of type `MPImage` using the whole image as region of interest. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageSegmenter` is created with `video`. This method supports segmentation of RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format. If your `MPImage` has a source type of `image` ensure that the color space is RGB with an Alpha channel. #### Declaration Swift func segment(videoFrame image: MPImage, timestampInMilliseconds: Int) throws -> ImageSegmenterResult #### Parameters |---|---| | `image` | The `MPImage` on which segmentation is to be performed. | | `timestampInMilliseconds` | The video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing. | #### Return Value An `ImageSegmenterResult` that contains a the segmented masks. - ` ### [segment(videoFrame:timestampInMilliseconds:completion:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageSegmenter.md.txt#/c:objc(cs)MPPImageSegmenter(im)segmentVideoFrame:timestampInMilliseconds:withCompletionHandler:) ` Performs segmentation on the provided video frame of type `MPImage` using the whole image as region of interest invokes the given completion handler block with the response. The method returns synchronously once the completion handler returns. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageSegmenter` is created with running mode, `video`. This method supports segmentation of RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format. If your `MPImage` has a source type of `image` ensure that the color space is RGB with an Alpha channel. #### Declaration Swift func segment(videoFrame image: MPImage, timestampInMilliseconds: Int, completion completionHandler: @escaping @Sendable (ImageSegmenterResult?, (any Error)?) -> Void) #### Parameters |---|---| | `image` | The `MPImage` on which segmentation is to be performed. | | `timestampInMilliseconds` | The video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing. | | `completionHandler` | A block to be invoked with the results of performing segmentation on the image. The block takes two arguments, the optional `ImageSegmenterResult` that contains the segmented masks if the segmentation was successful and an optional error only populated upon failure. The lifetime of the returned masks is only guaranteed for the duration of the block. | - ` ### [segmentAsync(image:timestampInMilliseconds:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageSegmenter.md.txt#/c:objc(cs)MPPImageSegmenter(im)segmentAsyncImage:timestampInMilliseconds:error:) ` Sends live stream image data of type `MPImage` to perform segmentation using the whole image as region of interest. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageSegmenter` is created with running mode, `liveStream`. The object which needs to be continuously notified of the available results of image segmentation must confirm to `ImageSegmenterLiveStreamDelegate` protocol and implement the `imageSegmenter(_:didFinishSegmentationWithResult:timestampInMilliseconds:error:)` delegate method. It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the segmenter. The input timestamps must be monotonically increasing. This method supports segmentation of RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format. If the input `MPImage` has a source type of `image` ensure that the color space is RGB with an Alpha channel. If this method is used for classifying live camera frames using `AVFoundation`, ensure that you request `AVCaptureVideoDataOutput` to output frames in `kCMPixelFormat_32BGRA` using its `videoSettings` property. #### Declaration Swift func segmentAsync(image: MPImage, timestampInMilliseconds: Int) throws #### Parameters |---|---| | `image` | A live stream image data of type `MPImage` on which segmentation is to be performed. | | `timestampInMilliseconds` | The timestamp (in milliseconds) which indicates when the input image is sent to the segmenter. The input timestamps must be monotonically increasing. | #### Return Value `YES` if the image was sent to the task successfully, otherwise `NO`.