Audio transcription guide for Android

The MediaPipe Audio Transcriber task lets you transcribe speech in audio data into text on the device. You can use this task to transcribe a live microphone stream or an audio clip. These instructions show you how to use the Audio Transcriber with Android apps.

For more information about the capabilities, models, and configuration options of this task, see the Overview.

Code example

The MediaPipe Tasks example code is a basic implementation of an Audio Transcriber app for Android. The example uses the microphone on a physical Android device to transcribe speech continuously. It shows interim results while you speak and final transcript segments with timestamps.

You can use the app as a starting point for your own Android app, or refer to it when modifying an existing app. The Audio Transcriber example code is hosted on GitHub.

Download the code

The following instructions show you how to create a local copy of the example code using the git command line tool.

To download the example code:

Clone the git repository using the following command:

git clone https://github.com/google-ai-edge/mediapipe-samples

Optionally, configure your git instance to use sparse checkout, so you have only the files for the Audio Transcriber example app:
```
cd mediapipe-samples
git sparse-checkout init --cone
git sparse-checkout set examples/audio_transcriber/android
```

After creating a local version of the example code, get the model as described in the example's README. Then import the project into Android Studio and run the app. For instructions, see the Setup guide for Android.

Key components

The following files contain the main code for the Audio Transcriber example app:

AudioTranscriberViewModel.kt - Creates the Audio Transcriber, streams microphone audio to it, and collects the interim and final transcript segments.
AudioCaptureEngine.kt - Captures 16 kHz mono audio from the microphone with AudioRecord.
MainActivity.kt - Implements the user interface and requests the microphone permission.

Setup

This section describes key steps for setting up your development environment and code projects specifically to use Audio Transcriber. For general information on setting up your development environment for using MediaPipe tasks, including platform version requirements, see the Setup guide for Android.

Dependencies

Audio Transcriber uses the com.google.mediapipe:tasks-audio library. Add this dependency to the build.gradle file of your Android app development project. Import the required dependencies with the following code:

dependencies {
    ...
    implementation 'com.google.mediapipe:tasks-audio:latest.release'
}

To transcribe the microphone, your app also needs the RECORD_AUDIO permission. Declare it in AndroidManifest.xml and request it at runtime:

<uses-permission android:name="android.permission.RECORD_AUDIO" />

Model

The MediaPipe Audio Transcriber task requires a speech recognition model that is compatible with this task. For more information on available models for Audio Transcriber, see the task overview Models section.

The model is too large to bundle in your APK. Download the model the first time your app needs it, store it in app-specific storage, and pass the absolute path of the file to the task:

File modelFile =
    new File(context.getFilesDir(), "parakeet_tdt_0.6b_v3_5s_i8_stateful.litertlm");

During development, you can push the model to a test device instead:

adb push parakeet_tdt_0.6b_v3_5s_i8_stateful.litertlm /data/local/tmp/

Use the BaseOptions.Builder.setModelAssetPath() method to specify the path to the model. This method is referred to in the code example in the next section.

Create the task

You can use the createFromOptions function to create the task. The createFromOptions function accepts configuration options including the model path, the input sample rate, and whether to return interim results. For more information on configuration options, see Configuration options.

import com.google.mediapipe.tasks.audio.audiotranscriber.AudioTranscriber;
import com.google.mediapipe.tasks.audio.audiotranscriber.AudioTranscriber.AudioTranscriberOptions;
import com.google.mediapipe.tasks.audio.audiotranscriber.AudioTranscriberResult;
import com.google.mediapipe.tasks.core.BaseOptions;

AudioTranscriberOptions options =
    AudioTranscriberOptions.builder()
        .setBaseOptions(
            BaseOptions.builder()
                .setModelAssetPath(modelFile.getAbsolutePath())
                .build())
        .setSampleRateHz(16000)
        .setEnableInterimResults(true)
        .build();
AudioTranscriber audioTranscriber = AudioTranscriber.createFromOptions(context, options);

If you don't need custom options, use AudioTranscriber.createFromFile(context, modelPath) instead.

Loading the model takes a moment. To get the first result faster, call prewarm() on a background thread before the user starts speaking:

backgroundExecutor.execute(audioTranscriber::prewarm);

Configuration options

This task has the following configuration options for Android apps:

Option Name	Description	Value Range	Default Value
`modelAssetPath`	Absolute path to the speech recognition model file (`.litertlm`) on the device. Set this option in the base options. You can also pass a file descriptor with `setModelAssetFileDescriptor`. In-memory model buffers aren't supported.	File path	None (required)
`delegate`	Hardware used to run the model. Set this option in the base options.	{`Delegate.CPU`, `Delegate.GPU`}	CPU
`sampleRateHz`	Sample rate of the input audio, in Hz. If the input isn't 16000 Hz, the task resamples it to 16 kHz before transcribing it.	Positive number	`16000`
`enableInterimResults`	Whether to return interim segments while audio is streaming. Interim segments are early hypotheses that can change as more audio arrives. The task later returns a final segment for the same audio.	Boolean	`false`
`modelArchitecture`	Model family. With the default value, the task detects the model family from the model file. Use the open-weights value for models such as Parakeet.	{`ModelArchitecture.AUTO`, `ModelArchitecture.OMNI`}	`ModelArchitecture.AUTO`
`cacheDir`	Writable directory where the task caches compiled model artifacts, so that later starts are faster.	Directory path	Not set

Prepare data

Audio Transcriber takes mono audio as an array of float samples in the range [-1.0, 1.0]. If your audio isn't 16 kHz, set sampleRateHz to its sample rate and the task resamples it.

Audio stream

import android.media.AudioFormat;
import android.media.AudioRecord;
import android.media.MediaRecorder;

int sampleRate = 16000;
int bufferSize =
    AudioRecord.getMinBufferSize(
        sampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_FLOAT);
AudioRecord audioRecord =
    new AudioRecord(
        MediaRecorder.AudioSource.VOICE_RECOGNITION,
        sampleRate,
        AudioFormat.CHANNEL_IN_MONO,
        AudioFormat.ENCODING_PCM_FLOAT,
        bufferSize);
audioRecord.startRecording();

// Read about 200 ms of audio at a time.
float[] chunk = new float[sampleRate / 5];

Audio clip

// Decode the audio file to mono float samples in [-1.0, 1.0], for example
// with MediaExtractor and MediaCodec. Set sampleRateHz in the task options to
// the sample rate of the decoded audio.
float[] samples = decodeToMonoFloat(audioFile);

Run the task

Call transcribe() with each chunk of audio. When the audio ends, call finishStream() to process any remaining audio and get the final segments. For a clip, pass all of the samples in one transcribe() call, and then call finishStream().

Audio stream

transcriptionExecutor.execute(() -> {
  while (isRecording) {
    int read = audioRecord.read(chunk, 0, chunk.length, AudioRecord.READ_BLOCKING);
    if (read > 0) {
      AudioTranscriberResult result =
          audioTranscriber.transcribe(Arrays.copyOf(chunk, read));
      showResult(result);
    }
  }
  audioRecord.stop();
  // Process the remaining audio and get the final segments.
  showResult(audioTranscriber.finishStream());
});

Audio clip

AudioTranscriberResult result = audioTranscriber.transcribe(samples);
AudioTranscriberResult finalResult = audioTranscriber.finishStream();

Note the following:

The transcribe(), finishStream(), and prewarm() methods block the current thread until they finish. Call them on a background thread, never on the main thread.
Keep transcription off your audio capture path. If transcription runs slower than real time on a device, read audio on one thread and hand the chunks to the transcription thread through a queue.
Each transcribe() call returns the segments that became available for that chunk. A call can return no segments.

Handle and display results

The Audio Transcriber returns an AudioTranscriberResult object for each call. The result contains a list of transcript segments. Each segment has its text, a flag that says whether it's final, and its start and end timestamps. The following listing shows an example of the output data from this task:

AudioTranscriberResult:
  Segment #0:
    transcript: "Turn on the kitchen lights"
    is final: true
    from timestamp (milliseconds): 0
    to timestamp (milliseconds): 1840
  Segment #1:
    transcript: "and set a timer for"
    is final: false
    from timestamp (milliseconds): 1840
    to timestamp (milliseconds): 2600

Final segments don't change. Interim segments are returned only when enableInterimResults is set, and the task replaces them with final segments as more audio arrives. One way to show live captions is to append final text and redraw the interim text after it:

private final StringBuilder finalText = new StringBuilder();

private void showResult(AudioTranscriberResult result) {
  StringBuilder interimText = new StringBuilder();
  for (AudioTranscriberResult.Segment segment : result.segments()) {
    if (segment.isFinal()) {
      finalText.append(segment.transcript()).append(' ');
    } else {
      interimText.append(segment.transcript()).append(' ');
    }
  }
  String caption = finalText.toString() + interimText;
  mainHandler.post(() -> captionView.setText(caption));
}

To get the combined text of all segments in a result, use result.getText().

Release resources

When you no longer need the task, call close() to release the model and its memory. AudioTranscriber implements AutoCloseable, so you can also create it in a try-with-resources statement:

try (AudioTranscriber audioTranscriber =
    AudioTranscriber.createFromOptions(context, options)) {
  // Transcribe audio.
}

Audio transcription guide for Android Stay organized with collections Save and categorize content based on your preferences.

Code example

Download the code

Key components

Setup

Dependencies

Model

Create the task

Configuration options

Prepare data

Audio stream

Audio clip

Run the task

Audio stream

Audio clip

Handle and display results

Release resources

Audio transcription guide for Android