Label images with an AutoML-trained model on iOS
After you train your own model using AutoML Vision Edge, you can use it in your app to label images.
There are two ways to integrate models trained from AutoML Vision Edge. You can bundle the model by copying the model's files into your Xcode project, or you can dynamically download it from Firebase.
Model bundling options | |
---|---|
Bundled in your app |
|
Hosted with Firebase |
|
Try it out
- Play around with the sample app to see an example usage of this API.
Before you begin
1. Include the ML Kit libraries in your Podfile:For bundling a model with your app:
pod 'GoogleMLKit/ImageLabelingAutoML'
LinkFirebase
dependency:
pod 'GoogleMLKit/ImageLabelingAutoML' pod 'GoogleMLKit/LinkFirebase'
.xcworkspace
code>. ML Kit is supported in Xcode
version 13.2.1 or greater.
3. If you want to download a model, make sure you
add Firebase to your iOS project,
if you have not already done so. This is not required when you bundle the
model.
1. Load the model
Configure a local model source
To bundle the model with your app:1. Extract the model and its metadata from the zip archive you downloaded from Firebase console into a folder:
your_model_directory |____dict.txt |____manifest.json |____model.tflite
2. Copy the folder to your Xcode project, taking care to select Create folder references when you do so. The model file and metadata will be included in the app bundle and available to ML Kit.
3. Create
AutoMLImageLabelerLocalModel
object, specifying the path to the
model manifest file:
Swift
guard let manifestPath = Bundle.main.path( forResource: "manifest", ofType: "json", inDirectory: "your_model_directory" ) else { return } let localModel = AutoMLImageLabelerLocalModel(manifestPath: manifestPath)
Objective-C
NSString *manifestPath = [NSBundle.mainBundle pathForResource:@"manifest" ofType:@"json" inDirectory:@"your_model_directory"]; MLKAutoMLImageLabelerLocalModel *localModel = [[MLKAutoMLImageLabelerLocalModel alloc] initWithManifestPath:manifestPath];
Configure a Firebase-hosted model source
To use the remotely-hosted model, create an AutoMLImageLabelerRemoteModel
object, specifying the name you assigned the model when you published it:
Swift
let remoteModel = AutoMLImageLabelerRemoteModel( name: "your_remote_model" // The name you assigned in // the Firebase console. )
Objective-C
MLKAutoMLImageLabelerRemoteModel *remoteModel = [[MLKAutoMLImageLabelerRemoteModel alloc] initWithName:@"your_remote_model"]; // The name you assigned in // the Firebase console.
Then, start the model download task, specifying the conditions under which you want to allow downloading. If the model isn't on the device, or if a newer version of the model is available, the task will asynchronously download the model from Firebase:
Swift
let downloadConditions = ModelDownloadConditions( allowsCellularAccess: true, allowsBackgroundDownloading: true ) let downloadProgress = ModelManager.modelManager().download( remoteModel, conditions: downloadConditions )
Objective-C
MLKModelDownloadConditions *downloadConditions = [[MLKModelDownloadConditions alloc] initWithAllowsCellularAccess:YES allowsBackgroundDownloading:YES]; NSProgress *downloadProgress = [[MLKModelManager modelManager] downloadModel:remoteModel conditions:downloadConditions];
Many apps start the download task in their initialization code, but you can do so at any point before you need to use the model.
Create an image labeler from your model
After you configure your model sources, create an ImageLabeler
object from one
of them.
If you only have a locally-bundled model, just create a labeler from your
AutoMLImageLabelerLocalModel
object and configure the confidence score
threshold you want to require (see Evaluate your mode:
Swift
let options = AutoMLImageLabelerOptions(localModel: localModel) options.confidenceThreshold = NSNumber(value: 0.0) // Evaluate your model in the Firebase console // to determine an appropriate value. let imageLabeler = ImageLabeler.imageLabeler(options: options)
Objective-C
MLKAutoMLImageLabelerOptions *options = [[MLKAutoMLImageLabelerOptions alloc] initWithLocalModel:localModel]; options.confidenceThreshold = @(0.0); // Evaluate your model in the Firebase console // to determine an appropriate value. MLKImageLabeler *imageLabeler = [MLKImageLabeler imageLabelerWithOptions:options];
If you have a remotely-hosted model, you will have to check that it has been
downloaded before you run it. You can check the status of the model download
task using the model manager's isModelDownloaded
(remoteModel:) method.
Although you only have to confirm this before running the labeler, if you
have both a remotely-hosted model and a locally-bundled model, it might make
sense to perform this check when instantiating the ImageLabeler
: create a
labeler from the remote model if it's been downloaded, and from the local model
otherwise.
Swift
var options: AutoMLImageLabelerOptions! if (ModelManager.modelManager().isModelDownloaded(remoteModel)) { options = AutoMLImageLabelerOptions(remoteModel: remoteModel) } else { options = AutoMLImageLabelerOptions(localModel: localModel) } options.confidenceThreshold = NSNumber(value: 0.0) // Evaluate your model in the Firebase console // to determine an appropriate value. let imageLabeler = ImageLabeler.imageLabeler(options: options)
Objective-C
MLKAutoMLImageLabelerOptions *options; if ([[MLKModelManager modelManager] isModelDownloaded:remoteModel]) { options = [[MLKAutoMLImageLabelerOptions alloc] initWithRemoteModel:remoteModel]; } else { options = [[MLKAutoMLImageLabelerOptions alloc] initWithLocalModel:localModel]; } options.confidenceThreshold = @(0.0); // Evaluate your model in the Firebase console // to determine an appropriate value. MLKImageLabeler *imageLabeler = [MLKImageLabeler imageLabelerWithOptions:options];
If you only have a remotely-hosted model, you should disable model-related functionality—for example, gray-out or hide part of your UI—until you confirm the model has been downloaded.
You can get the model download status by attaching observers to the default
Notification Center. Be sure to use a weak reference to self
in the observer
block, since downloads can take some time, and the originating object can be
freed by the time the download finishes. For example:
Swift
NotificationCenter.default.addObserver( forName: .mlkitModelDownloadDidSucceed, object: nil, queue: nil ) { [weak self] notification in guard let strongSelf = self, let userInfo = notification.userInfo, let model = userInfo[ModelDownloadUserInfoKey.remoteModel.rawValue] as? RemoteModel, model.name == "your_remote_model" else { return } // The model was downloaded and is available on the device } NotificationCenter.default.addObserver( forName: .mlkitModelDownloadDidFail, object: nil, queue: nil ) { [weak self] notification in guard let strongSelf = self, let userInfo = notification.userInfo, let model = userInfo[ModelDownloadUserInfoKey.remoteModel.rawValue] as? RemoteModel else { return } let error = userInfo[ModelDownloadUserInfoKey.error.rawValue] // ... }
Objective-C
__weak typeof(self) weakSelf = self; [NSNotificationCenter.defaultCenter addObserverForName:MLKModelDownloadDidSucceedNotification object:nil queue:nil usingBlock:^(NSNotification *_Nonnull note) { if (weakSelf == nil | note.userInfo == nil) { return; } __strong typeof(self) strongSelf = weakSelf; MLKRemoteModel *model = note.userInfo[MLKModelDownloadUserInfoKeyRemoteModel]; if ([model.name isEqualToString:@"your_remote_model"]) { // The model was downloaded and is available on the device } }]; [NSNotificationCenter.defaultCenter addObserverForName:MLKModelDownloadDidFailNotification object:nil queue:nil usingBlock:^(NSNotification *_Nonnull note) { if (weakSelf == nil | note.userInfo == nil) { return; } __strong typeof(self) strongSelf = weakSelf; NSError *error = note.userInfo[MLKModelDownloadUserInfoKeyError]; }];
2. Prepare the input image
Create a VisionImage
object using a UIImage
or a
CMSampleBuffer
.
If you use a UIImage
, follow these steps:
- Create a
VisionImage
object with theUIImage
. Make sure to specify the correct.orientation
.Swift
let image = VisionImage(image: UIImage) visionImage.orientation = image.imageOrientation
Objective-C
MLKVisionImage *visionImage = [[MLKVisionImage alloc] initWithImage:image]; visionImage.orientation = image.imageOrientation;
If you use a
CMSampleBuffer
, follow these steps:-
Specify the orientation of the image data contained in the
CMSampleBuffer
.To get the image orientation:
Swift
func imageOrientation( deviceOrientation: UIDeviceOrientation, cameraPosition: AVCaptureDevice.Position ) -> UIImage.Orientation { switch deviceOrientation { case .portrait: return cameraPosition == .front ? .leftMirrored : .right case .landscapeLeft: return cameraPosition == .front ? .downMirrored : .up case .portraitUpsideDown: return cameraPosition == .front ? .rightMirrored : .left case .landscapeRight: return cameraPosition == .front ? .upMirrored : .down case .faceDown, .faceUp, .unknown: return .up } }
Objective-C
- (UIImageOrientation) imageOrientationFromDeviceOrientation:(UIDeviceOrientation)deviceOrientation cameraPosition:(AVCaptureDevicePosition)cameraPosition { switch (deviceOrientation) { case UIDeviceOrientationPortrait: return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationLeftMirrored : UIImageOrientationRight; case UIDeviceOrientationLandscapeLeft: return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationDownMirrored : UIImageOrientationUp; case UIDeviceOrientationPortraitUpsideDown: return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationRightMirrored : UIImageOrientationLeft; case UIDeviceOrientationLandscapeRight: return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationUpMirrored : UIImageOrientationDown; case UIDeviceOrientationUnknown: case UIDeviceOrientationFaceUp: case UIDeviceOrientationFaceDown: return UIImageOrientationUp; } }
- Create a
VisionImage
object using theCMSampleBuffer
object and orientation:Swift
let image = VisionImage(buffer: sampleBuffer) image.orientation = imageOrientation( deviceOrientation: UIDevice.current.orientation, cameraPosition: cameraPosition)
Objective-C
MLKVisionImage *image = [[MLKVisionImage alloc] initWithBuffer:sampleBuffer]; image.orientation = [self imageOrientationFromDeviceOrientation:UIDevice.currentDevice.orientation cameraPosition:cameraPosition];
3. Run the image labeler
Asynchronously:
Swift
imageLabeler.process(image) { labels, error in guard error == nil, let labels = labels, !labels.isEmpty else { // Handle the error. return } // Show results. }
Objective-C
[imageLabeler processImage:image completion:^(NSArray
*_Nullable labels, NSError *_Nullable error) { if (labels.count == 0) { // Handle the error. return; } // Show results. }]; Synchronously:
Swift
var labels: [ImageLabel] do { labels = try imageLabeler.results(in: image) } catch let error { // Handle the error. return } // Show results.
Objective-C
NSError *error; NSArray
*labels = [imageLabeler resultsInImage:image error:&error]; // Show results or handle the error. 4. Get information about labeled objects
If the image labeling operation succeeds, it returns an array ofImageLabel
. EachImageLabel
represents something that was labeled in the image. You can get each label's text description (if available in the metadata of the TensorFlow Lite model file), confidence score, and index. For example:Swift
for label in labels { let labelText = label.text let confidence = label.confidence let index = label.index }
Objective-C
for (MLKImageLabel *label in labels) { NSString *labelText = label.text; float confidence = label.confidence; NSInteger index = label.index; }
Tips to improve real-time performance
If you want to label images in a real-time application, follow these guidelines to achieve the best framerates:
- For processing video frames, use the
results(in:)
synchronous API of the detector. Call this method from theAVCaptureVideoDataOutputSampleBufferDelegate
'scaptureOutput(_, didOutput:from:)
function to synchronously get results from the given video frame. KeepAVCaptureVideoDataOutput
'salwaysDiscardsLateVideoFrames
astrue
to throttle calls to the detector. If a new video frame becomes available while the detector is running, it will be dropped. - If you use the output of the detector to overlay graphics on the input image, first get the result from ML Kit, then render the image and overlay in a single step. By doing so, you render to the display surface only once for each processed input frame. See the updatePreviewOverlayViewWithLastFrame in the ML Kit quickstart sample for an example.
- For processing video frames, use the
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2024-10-04 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2024-10-04 UTC."],[],[]] -