GenAI Speech Recognition API

This API is offered in alpha, and is not subject to any SLA or deprecation policy. Changes may be made to this API that break backward compatibility.

Within ML Kit's GenAI Speech Recognition API, you can transcribe audio content to text. This API supports the following modes:

• Advanced: The Speech Recognition API uses the GenAI model, which produces broader language coverage and better overall quality. You don't have to download a separate language pack for each language, which saves on storage.
• Basic: The Speech Recognition API uses the traditional on-device speech recognition model, similar to the SpeechRecognizer API.

The Speech Recognition API transcribes audio content to text in the same language. To transcribe audio to text in a different language, use the Speech Translation API.

Key capabilities

• Captures streaming input from the microphone or an audio file.
• Transcribes a whole pre-recorded file with the AudioSource.fromFile() or AudioSource.fromUri() methods. For details, see Audio input requirements.
• For streaming inputs, provides the exact start time of each recognized word to help you synchronize UI animations, generate captions, or highlight words as they are spoken. The start time (in milliseconds) of each recognized word is an offset from the start of the event, which is timestamp 0.
• Transcribed text is provided as a continuous stream, which may initially be partial (and subject to change) before becoming the final recognized text.

Example results

Audio	Mode	Locale	Transcription
audio_2	Advanced	es-ES	"Este es un mensaje corto."
audio_1	Basic	en-US	"This is a short message"

Comparison with the platform Speech Recognition API

The ML Kit Speech Recognition API covers the basic functionality of the platform SpeechRecognizer API when using Basic mode. A key advantage of ML Kit is that it's not bound to a specific platform version.

Also, the ML Kit Speech Recognition API supports speech recognition using the on-device Gemini model when using Advanced mode, providing extended language coverage.

Note: Advanced mode is available on select devices. See Supported devices for details.

Get started

Note: The SDK requires minSdkVersion 26 or higher. See Supported devices for runtime requirements for each mode.

Add the ML Kit Speech Recognition API as a dependency in your build.gradle configuration:

implementation("com.google.mlkit:genai-speech-recognition:1.0.0-alpha1")

To integrate the Speech Recognition API into your app, create a SpeechRecognizer client. Check the status of the necessary on-device model features and download the model if it isn't already on the device. After preparing your audio input in a SpeechRecognizerRequest, run inference using the client to receive streaming output from the Kotlin flow. Finally, remember to close the client to release resources.

// 1. Create a SpeechRecognizer with desired options.
val options: SpeechRecognizerOptions =
    speechRecognizerOptions {
        locale = Locale.US
        preferredMode = SpeechRecognizerOptions.Mode.MODE_ADVANCED
    }
val speechRecognizer: SpeechRecognizer = SpeechRecognition.getClient(options)

// 2. Check if the recognition model is available or needs downloading.
launch {
    val status: Int = speechRecognizer.checkStatus()
    if (status == FeatureStatus.DOWNLOADABLE) {
        // 3. If needed, download the model and monitor progress.
        speechRecognizer.download.collect { downloadStatus ->
            when (downloadStatus) {
                is DownloadStatus.DownloadCompleted -> {
                    // Model is ready, start recognition.
                    startMyRecognition(speechRecognizer)
                }
                is DownloadStatus.DownloadFailed -> {
                    // Handle download failure (e.g., inform the user).
                }
                is DownloadStatus.DownloadProgress -> {
                    // Handle download progress (e.g., update a progress bar).
                }
            }
        }
    } else if (status == FeatureStatus.AVAILABLE) {
        // Model is already ready, start recognition immediately.
        startMyRecognition(speechRecognizer)
    } else {
        // Handle other statuses (e.g., DOWNLOADING, UNAVAILABLE).
    }
}

// 4. Define your recognition logic using a suspend function.
suspend fun startMyRecognition(recognizer: SpeechRecognizer) {
    // Create a request (e.g., specifying audio source).
    val request: SpeechRecognizerRequest
        = speechRecognizerRequest { audioSource = AudioSource.fromMic() }
    // Start recognition and process the continuous stream of responses.
    recognizer.startRecognition(request).collect {
        // Process the SpeechRecognitionResponse data here.
    }
}

// 5. Stop recognition and clean up resources when the session is complete.
launch {
    recognizer.stopRecognition()
    recognizer.close()
}

Audio input requirements

The GenAI Speech Recognition API supports input from the microphone or a custom source via a file descriptor.

If you use AudioSource.fromPfd(parcelFileDescriptor), the input audio must meet the following strict requirements:

• Format: Raw, headerless 16-bit PCM.
• Channels: Mono (single channel).
• Sample Rate: 16 kHz.

For most use cases, AudioSource.fromMic() is recommended as it handles these constraints automatically.

Supported audio input formats

The Speech Recognition API supports the following audio input formats:

• MP3, WAV, FLAC, M4A, and OGG files are decoded automatically on the device.
• Raw, headerless audio (.raw or .pcm) must be in 16-bit PCM, 16 kHz, mono format.

If the format is invalid or unsupported, an ErrorResponse value is returned when recognition starts.

Implement the Speech Recognition API

Here's how to use the Speech Recognition API to transcribe audio and use word timing.

Transcribe audio

The Speech Recognition API supports audio input from the device microphone, pre-recorded files, or custom pipelines using a ParcelFileDescriptor object.

To prepare your audio data, construct an AudioSource object using one of the factory methods listed in the following table:

Method	Audio mode	Recommended use case
AudioSource.fromMic()	STREAMING	Capture live voice input from the user. Requires the RECORD_AUDIO permission.
AudioSource.fromFile(file) AudioSource.fromUri(context, uri)	ONE_SHOT	Transcribe pre-recorded audio files. Automatically decodes standard file formats directly on the device.
AudioSource.fromPfd(pfd, mode) AudioSource.fromPfd(pfd, mode, audioFormat)	STREAMING or ONE_SHOT	Custom audio pipelines (such as audio data processed elsewhere in your app).

The AudioSource.Mode constant controls how the API processes the incoming audio:

• The AudioSource.Mode.STREAMING constant configures the recognizer to return partial and final transcription results continuously as the audio data arrives. When you provide a custom ParcelFileDescriptor object, you must write data to the stream at a real-time pace (approximately 32,000 bytes per second for standard 16 kHz, 16-bit, mono PCM audio). Writing data too quickly can overflow the buffer.
• The AudioSource.Mode.ONE_SHOT constant configures the recognizer to wait for the entire audio source to transfer before performing inference and delivering the final transcription. You can write the audio data as fast as possible, but you must close the write end of your file descriptor to signal end-of-file. The final transcription result is not delivered until this end-of-file signal is received.

Note: The AudioSource object takes ownership of the underlying ParcelFileDescriptor object. Remember to call the AudioSource.close() method when recognition is done.

The following example transcribes a pre-recorded audio file:

suspend fun recognizeFromFile(recognizer: SpeechRecognizer, audioFile: File) {
    val audioSource = AudioSource.fromFile(audioFile)
    val request = speechRecognizerRequest { this.audioSource = audioSource }
    try {
        recognizer.startRecognition(request).collect { response ->
            when (response) {
                is SpeechRecognizerResponse.FinalTextResponse -> {
                    // Process the transcription of the whole file.
                }
                is SpeechRecognizerResponse.ErrorResponse -> {
                    // Handle the error, for example an unsupported audio format.
                }
                else -> {}
            }
        }
    } finally {
        audioSource.close()
    }
}

Use word timing

You can request the start time of each recognized word by setting the enableWordTiming parameter to true in the recognitionOptions configuration block of your request.

When word timing is enabled, each SpeechRecognizerResponse.FinalTextResponse object includes a wordTimings property containing a list of SpeechRecognizerResponse.WordTiming objects. Each object provides the recognized word string (text) and its millisecond offset (timestampMs) starting from the beginning of the speech recognition session. When word timing is disabled (the default setting), the wordTimings list is returned empty.

The following code example demonstrates how to configure and enable word timing:

suspend fun recognizeWithWordTiming(recognizer: SpeechRecognizer) {
    val request = speechRecognizerRequest {
        audioSource = AudioSource.fromMic()
        recognitionOptions { enableWordTiming = true }
    }
    recognizer.startRecognition(request).collect { response ->
        if (response is SpeechRecognizerResponse.FinalTextResponse) {
            for (wordTiming in response.wordTimings) {
                val word: String = wordTiming.text
                val timestamp: Long = wordTiming.timestampMs
            }
        }
    }
}

Note: The word timing feature is supported only when using streaming audio sources, such as live input from the AudioSource.fromMic() method or custom streaming pipelines from the AudioSource.fromPfd() method with the AudioSource.Mode.STREAMING constant. Calling the startRecognition() method with word timing enabled on a static ONE_SHOT audio source, such as those returned by the AudioSource.fromFile() or AudioSource.fromUri() methods, throws an IllegalArgumentException object.

Supported languages

Mode	Locales
Advanced	ar-SA (beta), cmn-Hans-CN, cmn-Hant-TW, da-DK (beta), de-DE, en-US, es-ES, fr-FR, hi-IN (beta), id-ID (beta), it-IT, ja-JP, ko-KR, nl-NL (beta), pl-PL (beta), pt-PT, ru-RU, sv-SE (beta), th-TH, tr-TR (beta), vi-VN (beta)
Basic	cmn-Hans-CN (beta), cmn-Hant-TW (beta), de-DE (beta), en-US, es-ES (beta), fr-FR (beta), hi-IN (beta), it-IT (beta), ja-JP (beta), ko-KR (beta), pl-PL (beta), pt-BR (beta), ru-RU (beta), tr-TR (beta), vi-VN (beta)

Note: Languages in Beta may have a slightly higher word error rate. Additional locales are available. Evaluate their performance based on your specific use case.

Supported devices

Mode	Supported Devices
Basic	Android devices using API level 31 and higher.
Advanced	Pixel 10 and Pixel 11 series, plus devices with Gemini Nano on the following chipsets (including the Samsung Galaxy S26, Z Flip8, and Z Fold8 series, among others): Qualcomm Snapdragon: SM8750, SM8845, SM8850, SM8950, SM8975 MediaTek Dimensity: MT6991, MT6993, MT6995 Samsung Exynos: Exynos 2600

Common setup issues

Note: This API is not supported on devices with an unlocked bootloader.

ML Kit GenAI APIs rely on the Android AICore app to access Gemini Nano. When a device is just setup (including reset), or the AICore app is just reset (e.g. clear data, uninstalled then reinstalled), the AICore app may not have enough time to finish initialization (including downloading latest configurations from server). As a result, the ML Kit GenAI APIs may not function as expected. Here are the common setup error messages you may see and how to handle them:

Example error message	How to handle
AICore failed with error type 4-CONNECTION_ERROR and error code 601-BINDING_FAILURE: AICore service failed to bind.	This could happen when you install the app using ML Kit GenAI APIs immediately after device setup or when AICore is uninstalled after your app is installed. Updating AICore app then reinstalling your app should fix it.
AICore failed with error type 3-PREPARATION_ERROR and error code 606-FEATURE_NOT_FOUND: Feature ... is not available.	This could happen when AICore hasn't finished downloading the latest configurations. When the device is connected to the internet, it usually takes a few minutes to a few hours to update. Restarting the device can speed up the update. Note that if the device's bootloader is unlocked, you'll also see this error—this API does not support devices with unlocked bootloaders.
AICore failed with error type 1-DOWNLOAD_ERROR and error code 0-UNKNOWN: Feature ... failed with failure status 0 and error esz: UNAVAILABLE: Unable to resolve host ...	Keep network connection, wait for a few minutes and retry.

Note: While various resetting is common to developers, it is much less frequent to users. Also, all these setup failure cases can be caught by the checkFeatureStatus() or checkStatus() APIs. Make sure to call checkFeatureStatus() or checkStatus() first before showing any related UI, so your app users won't see these errors in any cases.

Sample code

• Explore the ML Kit Speech Recognition API code sample on GitHub