Audio transcription guide for Python

The MediaPipe Audio Transcriber task lets you transcribe speech in audio data into text on the device. You can use this task to transcribe an audio file or a live audio stream. These instructions show you how to use the Audio Transcriber with Python.

For more information about the capabilities, models, and configuration options of this task, see the Overview.

Code example

The example code for Audio Transcriber provides a complete implementation of this task in Python for your reference. The example transcribes a speech recording in a single call, and then streams the same recording in 1-second chunks, showing interim and final results as they arrive. You can view, run, and edit the Audio Transcriber example code using just your web browser with Google Colab. You can view the source code for this example, including a command-line version, on GitHub.

Setup

This section describes key steps for setting up your development environment and code projects specifically to use Audio Transcriber. For general information on setting up your development environment for using MediaPipe tasks, including platform version requirements, see the Setup guide for Python.

Packages

The Audio Transcriber task uses the mediapipe pip package. You can install the dependency with the following:

$ python -m pip install mediapipe numpy

Imports

Import the following modules to access the Audio Transcriber task functions:

from mediapipe.tasks.python.audio import audio_transcriber
from mediapipe.tasks.python.core import base_options

Model

The MediaPipe Audio Transcriber task requires a speech recognition model that is compatible with this task. For more information on available models for Audio Transcriber, see the task overview Models section.

Download the model, store it in a local directory, and keep its original filename:

model_path = '/absolute/path/to/parakeet_tdt_0.6b_v3_5s_i8_stateful.litertlm'

Specify the path of the model in the base options, as shown in the next section.

Create the task

Use the create_from_options function to create the task. The create_from_options function accepts configuration options including the model path, the delegate, the input sample rate, and whether to return interim results. For more information on configuration options, see Configuration options.

BaseOptions = base_options.BaseOptions
AudioTranscriber = audio_transcriber.AudioTranscriber
AudioTranscriberOptions = audio_transcriber.AudioTranscriberOptions

options = AudioTranscriberOptions(
    base_options=BaseOptions(
        model_asset_path=model_path,
        delegate=BaseOptions.Delegate.CPU),
    sample_rate_hz=16000,
    enable_interim_results=True)

with AudioTranscriber.create_from_options(options) as transcriber:
  # The transcriber is initialized. Use it here.
  # ...

If you don't need custom options, use AudioTranscriber.create_from_model_path(model_path) instead.

Loading the model takes a moment. To get the first result faster, call prewarm() before you start transcribing:

transcriber.prewarm()

To run the model on the GPU, set the delegate to BaseOptions.Delegate.GPU. If the GPU isn't available, create_from_options raises a RuntimeError, and you can create the task with BaseOptions.Delegate.CPU instead. On Linux, the GPU delegate uses Vulkan, and the user running your application needs access to the GPU device, for example by being a member of the render group.

For a complete example of creating an Audio Transcriber, see the code example.

Configuration options

This task has the following configuration options for Python applications:

Option Name Description Value Range Default Value
model_asset_path Absolute path to the speech recognition model file (.litertlm) on the device. Set this option in the base options. In-memory model buffers aren't supported. File path None (required)
delegate Hardware used to run the model. Set this option in the base options. {BaseOptions.Delegate.CPU, BaseOptions.Delegate.GPU} CPU
sample_rate_hz Sample rate of the input audio, in Hz. If the input isn't 16000 Hz, the task resamples it to 16 kHz before transcribing it. Positive number 16000
enable_interim_results Whether to return interim segments while audio is streaming. Interim segments are early hypotheses that can change as more audio arrives. The task later returns a final segment for the same audio. Boolean False
model_architecture Model family. With the default value, the task detects the model family from the model file. Use the open-weights value for models such as Parakeet. {ModelArchitecture.AUTO, ModelArchitecture.OMNI} ModelArchitecture.AUTO
cache_dir Writable directory where the task caches compiled model artifacts, so that later starts are faster. Directory path Not set

Prepare data

Audio Transcriber takes mono audio as a NumPy array of float32 samples in the range [-1.0, 1.0]. If your audio isn't 16 kHz, set sample_rate_hz to its sample rate and the task resamples it.

The following examples explain and show how to prepare data for processing for each of the available data types:

Audio clip

import wave

import numpy as np

# Read a 16-bit PCM .wav file and convert it to mono float32 samples.
with wave.open('/path/to/audio.wav', 'rb') as wav_file:
  sample_rate = wav_file.getframerate()
  num_channels = wav_file.getnchannels()
  frames = wav_file.readframes(wav_file.getnframes())

samples = np.frombuffer(frames, dtype=np.int16).astype(np.float32) / 32768
samples = samples.reshape(-1, num_channels).mean(axis=1)

Audio stream

import numpy as np

# Read microphone data as mono float32 NumPy arrays, for example 200 ms
# at a time, then convert integer samples to the range [-1.0, 1.0]:
chunk = buffer.astype(np.float32) / np.iinfo(np.int16).max

Run the task

Call transcribe() to trigger inference. For an audio clip, pass all of the samples in one call with end_of_stream=True. For an audio stream, call transcribe() with each chunk of audio and end_of_stream=False. When the audio ends, call finish_stream() to process any remaining audio and get the final segments.

Audio clip

# Transcribe the whole clip in a single call.
result = transcriber.transcribe(samples, end_of_stream=True)

Audio stream

# Transcribe live audio chunk by chunk.
while is_recording:
  result = transcriber.transcribe(read_next_chunk(), end_of_stream=False)
  show_result(result)

# Process the remaining audio and get the final segments.
show_result(transcriber.finish_stream())

Note the following:

  • The transcribe(), finish_stream(), and prewarm() functions block the current thread until they finish. In an application that captures audio with a callback, keep the callback short: put the audio chunks on a queue and transcribe them on another thread.
  • When running on an audio stream, each transcribe() call returns the segments that became available for that chunk. A call can return no segments.
  • After a call with end_of_stream=True or a call to finish_stream(), the next transcribe() call starts a new stream.

For a more complete example of running Audio Transcriber on an audio clip and an audio stream, see the code example.

Handle and display results

The Audio Transcriber returns an AudioTranscriberResult object for each call. The result contains a list of transcript segments. Each segment has its transcript, an is_final flag that says whether it's final, and its from_timestamp_ms and to_timestamp_ms timestamps. The following listing shows an example of the output data from this task:

AudioTranscriberResult:
  Segment #0:
    transcript: "Turn on the kitchen lights"
    is final: true
    from timestamp (milliseconds): 0
    to timestamp (milliseconds): 1840
  Segment #1:
    transcript: "and set a timer for"
    is final: false
    from timestamp (milliseconds): 1840
    to timestamp (milliseconds): 2600

Final segments don't change. Interim segments are returned only when enable_interim_results is set, and the task replaces them with final segments as more audio arrives. One way to show live captions is to append final text and redraw the interim text after it:

final_text = []

def show_result(result):
  interim_text = []
  for segment in result.segments:
    if segment.is_final:
      final_text.append(segment.transcript)
    else:
      interim_text.append(segment.transcript)
  print(' '.join(final_text + interim_text))

To get the combined text of all segments in a result, use result.text.

Release resources

When you no longer need the task, call close() to release the model and its memory. You can also use the task as a context manager in a with statement, as shown in Create the task, which closes it automatically.