You can use ML Kit to label objects recognized in an image. The default model provided with ML Kit supports 400+ different labels.
This API is available using either an unbundled library that must be downloaded before use or a bundled library that increases your app size. See [this guide](https://developers.google.com/ml-kit/tips/installation-paths.md.txt) for more information on the differences between the two installation options. | Feature | Unbundled | Bundled | |---|---|---| | Implementation | Model is dynamically downloaded via Google Play Services. | Model is statically linked to your at build time. | | App size | About 200 KB size increase. | About 5.7 MB size increase. | | Initialization time | Might have to wait for model to download before first use. | Model is available immediately | ## Try it out - Play around with [the sample app](https://github.com/googlesamples/mlkit/tree/master/android/vision-quickstart) to see an example usage of this API. ## Before you begin > [!NOTE] > This API requires Android API level 23 or above. Make sure that your app's build file uses a `minSdkVersion` value of 23 or higher. 1. In your project-level `build.gradle` file, make sure to include Google's Maven repository in both your `buildscript` and `allprojects` sections. 2. Add the dependencies for the ML Kit Android libraries to your module's app-level gradle file, which is usually `app/build.gradle`. Choose one of the following dependencies based on your needs: **For bundling the model with your app:** dependencies { // ... // Use this dependency to bundle the model with your app implementation 'com.google.mlkit:image-labeling:17.0.9' } **For using the model in Google Play Services:** dependencies { // ... // Use this dependency to use the dynamically downloaded model in Google Play Services implementation 'com.google.android.gms:play-services-mlkit-image-labeling:16.0.8' } 3. **If you choose to use the model in Google Play Services** , you can configure your app to automatically download the model to the device after your app is installed from the Play Store. To do so, add the following declaration to your app's `AndroidManifest.xml` file: ... You can also explicitly check the model availability and request download through Google Play services [ModuleInstallClient API](https://developers.google.com/android/guides/module-install-apis.md.txt). If you don't enable install-time model downloads or request explicit download, the model is downloaded the first time you run the labeler. Requests you make before the download has completed produce no results. Now you are ready to label images. ## 1. Prepare the input image Create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from your image. The image labeler runs fastest when you use a `Bitmap` or, if you use the camera2 API, a YUV_420_888 `media.Image`, which are recommended when possible.
You can create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from different sources, each is explained below. ### Using a `media.Image` To create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from a `media.Image` object, such as when you capture an image from a device's camera, pass the `media.Image` object and the image's rotation to `InputImage.fromMediaImage()`. If you use the [CameraX](https://developer.android.com/training/camerax) library, the `OnImageCapturedListener` and `ImageAnalysis.Analyzer` classes calculate the rotation value for you. ### Kotlin ```kotlin private class YourImageAnalyzer : ImageAnalysis.Analyzer { override fun analyze(imageProxy: ImageProxy) { val mediaImage = imageProxy.image if (mediaImage != null) { val image = InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees) // Pass image to an ML Kit Vision API // ... } } } ``` ### Java ```java private class YourAnalyzer implements ImageAnalysis.Analyzer { @Override public void analyze(ImageProxy imageProxy) { Image mediaImage = imageProxy.getImage(); if (mediaImage != null) { InputImage image = InputImage.fromMediaImage(mediaImage, imageProxy.getImageInfo().getRotationDegrees()); // Pass image to an ML Kit Vision API // ... } } } ``` If you don't use a camera library that gives you the image's rotation degree, you can calculate it from the device's rotation degree and the orientation of camera sensor in the device: ### Kotlin ```kotlin private val ORIENTATIONS = SparseIntArray() init { ORIENTATIONS.append(Surface.ROTATION_0, 0) ORIENTATIONS.append(Surface.ROTATION_90, 90) ORIENTATIONS.append(Surface.ROTATION_180, 180) ORIENTATIONS.append(Surface.ROTATION_270, 270) } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP) @Throws(CameraAccessException::class) private fun getRotationCompensation(cameraId: String, activity: Activity, isFrontFacing: Boolean): Int { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. val deviceRotation = activity.windowManager.defaultDisplay.rotation var rotationCompensation = ORIENTATIONS.get(deviceRotation) // Get the device's sensor orientation. val cameraManager = activity.getSystemService(CAMERA_SERVICE) as CameraManager val sensorOrientation = cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.SENSOR_ORIENTATION)!! if (isFrontFacing) { rotationCompensation = (sensorOrientation + rotationCompensation) % 360 } else { // back-facing rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360 } return rotationCompensation } ``` ### Java ```java private static final SparseIntArray ORIENTATIONS = new SparseIntArray(); static { ORIENTATIONS.append(Surface.ROTATION_0, 0); ORIENTATIONS.append(Surface.ROTATION_90, 90); ORIENTATIONS.append(Surface.ROTATION_180, 180); ORIENTATIONS.append(Surface.ROTATION_270, 270); } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP) private int getRotationCompensation(String cameraId, Activity activity, boolean isFrontFacing) throws CameraAccessException { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation(); int rotationCompensation = ORIENTATIONS.get(deviceRotation); // Get the device's sensor orientation. CameraManager cameraManager = (CameraManager) activity.getSystemService(CAMERA_SERVICE); int sensorOrientation = cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.SENSOR_ORIENTATION); if (isFrontFacing) { rotationCompensation = (sensorOrientation + rotationCompensation) % 360; } else { // back-facing rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360; } return rotationCompensation; } ``` Then, pass the `media.Image` object and the rotation degree value to `InputImage.fromMediaImage()`: ### Kotlin ```kotlin val image = InputImage.fromMediaImage(mediaImage, rotation) ``` ### Java ```java InputImage image = InputImage.fromMediaImage(mediaImage, rotation); ``` ### Using a file URI To create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from a file URI, pass the app context and file URI to `InputImage.fromFilePath()`. This is useful when you use an `ACTION_GET_CONTENT` intent to prompt the user to select an image from their gallery app. ### Kotlin ```kotlin val image: InputImage try { image = InputImage.fromFilePath(context, uri) } catch (e: IOException) { e.printStackTrace() } ``` ### Java ```java InputImage image; try { image = InputImage.fromFilePath(context, uri); } catch (IOException e) { e.printStackTrace(); } ``` ### Using a `ByteBuffer` or `ByteArray` To create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from a `ByteBuffer` or a `ByteArray`, first calculate the image rotation degree as previously described for `media.Image` input. Then, create the `InputImage` object with the buffer or array, together with image's height, width, color encoding format, and rotation degree: ### Kotlin ```kotlin val image = InputImage.fromByteBuffer( byteBuffer, /* image width */ 480, /* image height */ 360, rotationDegrees, InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12 ) // Or: val image = InputImage.fromByteArray( byteArray, /* image width */ 480, /* image height */ 360, rotationDegrees, InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12 ) ``` ### Java ```java InputImage image = InputImage.fromByteBuffer(byteBuffer, /* image width */ 480, /* image height */ 360, rotationDegrees, InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12 ); // Or: InputImage image = InputImage.fromByteArray( byteArray, /* image width */480, /* image height */360, rotation, InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12 ); ``` ### Using a `Bitmap` To create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from a `Bitmap` object, make the following declaration: ### Kotlin ```kotlin val image = InputImage.fromBitmap(bitmap, 0) ``` ### Java ```java InputImage image = InputImage.fromBitmap(bitmap, rotationDegree); ``` The image is represented by a `Bitmap` object together with rotation degrees. ## 2. Configure and run the image labeler To label objects in an image, pass the `InputImage` object to the `ImageLabeler`'s `process` method.
1. First, get an instance of [`ImageLabeler`](https://developers.google.com/android/reference/com/google/mlkit/vision/label/ImageLabeler.md.txt). If you want to use the on-device image labeler, make the following declaration: ### Kotlin ```kotlin // To use default options: val labeler = ImageLabeling.getClient(ImageLabelerOptions.DEFAULT_OPTIONS) // Or, to set the minimum confidence required: // val options = ImageLabelerOptions.Builder() // .setConfidenceThreshold(0.7f) // .build() // val labeler = ImageLabeling.getClient(options) ``` ### Java ```java // To use default options: ImageLabeler labeler = ImageLabeling.getClient(ImageLabelerOptions.DEFAULT_OPTIONS); // Or, to set the minimum confidence required: // ImageLabelerOptions options = // new ImageLabelerOptions.Builder() // .setConfidenceThreshold(0.7f) // .build(); // ImageLabeler labeler = ImageLabeling.getClient(options); ``` 1. Then, pass the image to the `process()` method: ### Kotlin ```kotlin labeler.process(image) .addOnSuccessListener { labels -> // Task completed successfully // ... } .addOnFailureListener { e -> // Task failed with an exception // ... } ``` ### Java ```java labeler.process(image) .addOnSuccessListener(new OnSuccessListener>() { @Override public void onSuccess(List labels) { // Task completed successfully // ... } }) .addOnFailureListener(new OnFailureListener() { @Override public void onFailure(@NonNull Exception e) { // Task failed with an exception // ... } }); ``` Note: If you are using the [`CameraX`](https://developer.android.com/jetpack/androidx/releases/camera) API, make sure to close the `ImageProxy` when finish using it, e.g., by adding an `OnCompleteListener` to the `Task` returned from the `process` method. See the [`VisionProcessorBase`](https://github.com/googlesamples/mlkit/blob/e949654bef09e97b237c7692349c52ee06e9fd9e/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/java/VisionProcessorBase.java#L178-L181) class in the quickstart sample app for an example. ## 3. Get information about labeled objects If the image labeling operation succeeds, a list of [`ImageLabel`](https://developers.google.com/android/reference/com/google/mlkit/vision/label/ImageLabel.md.txt) objects is passed to the success listener. Each `ImageLabel` object represents something that was labeled in the image. The base model supports [400+ different labels](https://developers.google.com/ml-kit/vision/image-labeling/label-map.md.txt). You can get each label's text description, index among all labels supported by the model, and the confidence score of the match. For example:
### Kotlin ```kotlin for (label in labels) { val text = label.text val confidence = label.confidence val index = label.index } ``` ### Java ```java for (ImageLabel label : labels) { String text = label.getText(); float confidence = label.getConfidence(); int index = label.getIndex(); } ``` ## Tips to improve real-time performance If you want to label images in a real-time application, follow these guidelines to achieve the best framerates: - If you use the [`Camera`](https://developer.android.com/reference/android/hardware/Camera) or [`camera2`](https://developer.android.com/reference/android/hardware/camera2/package-summary) API, throttle calls to the image labeler. If a new video frame becomes available while the image labeler is running, drop the frame. See the [`VisionProcessorBase`](https://github.com/googlesamples/mlkit/blob/master/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/kotlin/VisionProcessorBase.kt) class in the quickstart sample app for an example. - If you use the [`CameraX`](https://developer.android.com/jetpack/androidx/releases/camera) API, be sure that backpressure strategy is set to its default value [`ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST`](https://developer.android.com/reference/androidx/camera/core/ImageAnalysis#STRATEGY_KEEP_ONLY_LATEST). This guarantees only one image will be delivered for analysis at a time. If more images are produced when the analyzer is busy, they will be dropped automatically and not queued for delivery. Once the image being analyzed is closed by calling ImageProxy.close(), the next latest image will be delivered. - If you use the output of the image labeler to overlay graphics on the input image, first get the result from ML Kit, then render the image and overlay in a single step. This renders to the display surface only once for each input frame. See the [`CameraSourcePreview`](https://github.com/googlesamples/mlkit/blob/master/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/CameraSourcePreview.java) and [`GraphicOverlay`](https://github.com/googlesamples/mlkit/blob/master/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/GraphicOverlay.java) classes in the quickstart sample app for an example. - If you use the Camera2 API, capture images in `ImageFormat.YUV_420_888` format. If you use the older Camera API, capture images in `ImageFormat.NV21` format.