# ImageEmbedder

    class ImageEmbedder: NSObject, EmbeddingProvider

Performs embedding extraction on images.

The API expects a TFLite model with optional, but strongly recommended, \[TFLite Model Metadata.\](https://www.tensorflow.org/lite/convert/metadata").

The API supports models with one image input tensor and one or more output tensors. To be more specific, here are the requirements.

Input image tensor (kTfLiteUInt8/kTfLiteFloat32) - image input of size `[batch x height x width x channels]`. - batch inference is not supported (`batch` is required to be 1). - only RGB inputs are supported (`channels` is required to be 3). - if type is kTfLiteFloat32, NormalizationOptions are required to be attached to the metadata for input normalization.

At least one output tensor (kTfLiteUInt8/kTfLiteFloat32) with shape `[1 x N]` where N is the number of dimensions in the produced embeddings.
- `
  ### [cosineSimilarity(embedding1:embedding2:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageEmbedder.md.txt#/c:objc(cs)MPPImageEmbedder(cm)cosineSimilarityBetweenEmbedding1:andEmbedding2:error:)

  ` Utility function to compute\[cosine similarity\](https://en.wikipedia.org/wiki/Cosine_similarity) between two `MPPEmbedding` objects.

  #### Declaration

  Swift

      class func cosineSimilarity(embedding1: Embedding, embedding2: Embedding) throws -> NSNumber

  #### Parameters

  |---|---|
  | `embedding1` | One of the two `MPPEmbedding`s between whom cosine similarity is to be computed. |
  | `embedding2` | One of the two `MPPEmbedding`s between whom cosine similarity is to be computed. |
  | `error` | An optional error parameter populated when there is an error in calculating cosine similarity between two embeddings. |

  #### Return Value

  An `NSNumber` which holds the cosine similarity of type `double`.
- `
  ### [init(modelPath:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageEmbedder.md.txt#/c:objc(cs)MPPImageEmbedder(im)initWithModelPath:error:)

  ` Creates a new instance of `ImageEmbedder` from an absolute path to a TensorFlow Lite model file stored locally on the device and the default `ImageEmbedderOptions`.

  #### Declaration

  Swift

      convenience init(modelPath: String) throws

  #### Parameters

  |---|---|
  | `modelPath` | An absolute path to a TensorFlow Lite model file stored locally on the device. |

  #### Return Value

  A new instance of `ImageEmbedder` with the given model path. `nil` if there is an error in initializing the image embedder.
- `
  ### [init(options:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageEmbedder.md.txt#/c:objc(cs)MPPImageEmbedder(im)initWithOptions:error:)

  ` Creates a new instance of `ImageEmbedder` from the given `ImageEmbedderOptions`.

  #### Declaration

  Swift

      init(options: ImageEmbedderOptions) throws

  #### Parameters

  |---|---|
  | `options` | The options of type `ImageEmbedderOptions` to use for configuring the `ImageEmbedder`. |

  #### Return Value

  A new instance of `ImageEmbedder` with the given options. `nil` if there is an error in initializing the image embedder.
- `
  ### [embed(image:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageEmbedder.md.txt#/c:objc(cs)MPPImageEmbedder(im)embedImage:error:)

  ` Performs embedding extraction on the provided `MPImage` using the whole image as region of interest. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageEmbedder` is created with running mode, `.image`.

  This method supports embedding extraction on RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format.

  If your `MPImage` has a source type of `.image` ensure that the color space is RGB with an Alpha channel.

  #### Declaration

  Swift

      func embed(image: MPImage) throws -> ImageEmbedderResult

  #### Parameters

  |---|---|
  | `image` | The `MPImage` on which embedding extraction is to be performed. |

  #### Return Value

  An `ImageEmbedderResult` object that contains a list of embedding extraction.
- `
  ### [embed(image:regionOfInterest:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageEmbedder.md.txt#/c:objc(cs)MPPImageEmbedder(im)embedImage:regionOfInterest:error:)

  ` Performs embedding extraction on the provided `MPImage` cropped to the specified region of interest. Rotation will be applied on the cropped image according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageEmbedder` is created with running mode, `.image`.

  This method supports embedding extraction on RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format.

  If your `MPImage` has a source type of `.image` ensure that the color space is RGB with an Alpha channel.

  #### Declaration

  Swift

      func embed(image: MPImage, regionOfInterest roi: CGRect) throws -> ImageEmbedderResult

  #### Parameters

  |---|---|
  | `image` | The `MPImage` on which embedding extraction is to be performed. |
  | `roi` | A `CGRect` specifying the region of interest within the given `MPImage`, on which embedding extraction should be performed. |

  #### Return Value

  An `ImageEmbedderResult` object that contains a list of generated image embeddings.
- `
  ### [embed(videoFrame:timestampInMilliseconds:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageEmbedder.md.txt#/c:objc(cs)MPPImageEmbedder(im)embedVideoFrame:timestampInMilliseconds:error:)

  ` Performs embedding extraction on the provided video frame of type `MPImage` using the whole image as region of interest. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageEmbedder` is created with running mode `.video`.

  It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

  This method supports embedding extraction on RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format.

  If your `MPImage` has a source type of `.image` ensure that the color space is RGB with an Alpha channel.

  #### Declaration

  Swift

      func embed(videoFrame image: MPImage, timestampInMilliseconds: Int) throws -> ImageEmbedderResult

  #### Parameters

  |---|---|
  | `image` | The `MPImage` on which embedding extraction is to be performed. |
  | `timestampInMilliseconds` | The video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing. |

  #### Return Value

  An `ImageEmbedderResult` object that contains a list of generated image embeddings.
- `
  ### [embed(videoFrame:timestampInMilliseconds:regionOfInterest:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageEmbedder.md.txt#/c:objc(cs)MPPImageEmbedder(im)embedVideoFrame:timestampInMilliseconds:regionOfInterest:error:)

  ` Performs embedding extraction on the provided video frame of type `MPImage` cropped to the specified region of interest. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageEmbedder` is created with `.video`.

  It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

  This method supports embedding extraction on RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format.

  If your `MPImage` has a source type of `.image` ensure that the color space is RGB with an Alpha channel.

  #### Declaration

  Swift

      func embed(videoFrame image: MPImage, timestampInMilliseconds: Int, regionOfInterest roi: CGRect) throws -> ImageEmbedderResult

  #### Parameters

  |---|---|
  | `image` | A live stream image data of type `MPImage` on which embedding extraction is to be performed. |
  | `timestampInMilliseconds` | The video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing. |
  | `roi` | A `CGRect` specifying the region of interest within the video frame of type `MPImage`, on which embedding extraction should be performed. |

  #### Return Value

  An `ImageEmbedderResult` object that contains a list of generated image embeddings.
- `
  ### [embedAsync(image:timestampInMilliseconds:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageEmbedder.md.txt#/c:objc(cs)MPPImageEmbedder(im)embedAsyncImage:timestampInMilliseconds:error:)

  ` Sends live stream image data of type `MPImage` to perform embedding extraction using the whole image as region of interest. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageEmbedder` is created with running mode `.liveStream`.

  The object which needs to be continuously notified of the available results of image embedding extraction must confirm to `ImageEmbedderLiveStreamDelegate` protocol and implement the `imageEmbedder(_:didFinishEmbeddingWithResult:timestampInMilliseconds:error:)` delegate method.

  It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the image embedder. The input timestamps must be monotonically increasing.

  This method supports embedding extraction on RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format.

  If the input `MPImage` has a source type of `.image` ensure that the color space is RGB with an Alpha channel.

  If this method is used for embedding live camera frames using `AVFoundation`, ensure that you request `AVCaptureVideoDataOutput` to output frames in `kCMPixelFormat_32BGRA` using its `videoSettings` property.

  #### Declaration

  Swift

      func embedAsync(image: MPImage, timestampInMilliseconds: Int) throws

  #### Parameters

  |---|---|
  | `image` | A live stream image data of type `MPImage` on which embedding extraction is to be performed. |
  | `timestampInMilliseconds` | The timestamp (in milliseconds) which indicates when the input image is sent to the image embedder. The input timestamps must be monotonically increasing. |

  #### Return Value

  `true` if the image was sent to the task successfully, otherwise `false`.
- `
  ### [embedAsync(image:timestampInMilliseconds:regionOfInterest:)](https://developers.google.com/edge/api/mediapipe/swift/vision/Classes/ImageEmbedder.md.txt#/c:objc(cs)MPPImageEmbedder(im)embedAsyncImage:timestampInMilliseconds:regionOfInterest:error:)

  ` Sends live stream image data of type `MPImage` to perform embedding extraction, cropped to the specified region of interest.. Rotation will be applied according to the `orientation` property of the provided `MPImage`. Only use this method when the `ImageEmbedder` is created with `.liveStream`.

  The object which needs to be continuously notified of the available results of image embedding extraction must confirm to `ImageEmbedderLiveStreamDelegate` protocol and implement the `imageEmbedder(_:didFinishEmbeddingWithResult:timestampInMilliseconds:error:)` delegate method.

  It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the image embedder. The input timestamps must be monotonically increasing.

  This method supports embedding extraction on RGBA images. If your `MPImage` has a source type of `.pixelBuffer` or `.sampleBuffer`, the underlying pixel buffer must use `kCVPixelFormatType_32BGRA` as its pixel format.

  If the input `MPImage` has a source type of `.image` ensure that the color space is RGB with an Alpha channel.

  If this method is used for embedding live camera frames using `AVFoundation`, ensure that you request `AVCaptureVideoDataOutput` to output frames in `kCMPixelFormat_32BGRA` using its `videoSettings` property.

  #### Declaration

  Swift

      func embedAsync(image: MPImage, timestampInMilliseconds: Int, regionOfInterest roi: CGRect) throws

  #### Parameters

  |---|---|
  | `image` | A live stream image data of type `MPImage` on which embedding extraction is to be performed. |
  | `timestampInMilliseconds` | The timestamp (in milliseconds) which indicates when the input image is sent to the image embedder. The input timestamps must be monotonically increasing. |
  | `roi` | A `CGRect` specifying the region of interest within the given live stream image data of type `MPImage`, on which embedding extraction should be performed. |

  #### Return Value

  `true` if the image was sent to the task successfully, otherwise `false`.