Audio transcription guide for web

The MediaPipe Audio Transcriber task lets you transcribe speech in audio data into text in the browser. You can use this task to transcribe an audio file or a live audio stream, such as the microphone. These instructions show you how to use the Audio Transcriber for web apps.

You can see this task in action by viewing the demo. For more information about the capabilities, models, and configuration options of this task, see the Overview.

Code example

The example code for Audio Transcriber provides a complete implementation of this task in TypeScript for your reference. The example runs the task in a web worker, transcribes speech from the microphone with live interim results, and transcribes uploaded audio and video files. You can view, run, and edit the Audio Transcriber example using just your web browser. You can view the source code for this example on GitHub.

Setup

This section describes key steps for setting up your development environment and code projects specifically to use Audio Transcriber. For general information on setting up your development environment for using MediaPipe tasks, including platform version requirements, see the Setup guide for web.

JavaScript packages

Audio Transcriber code is available through the MediaPipe @mediapipe/tasks-audio NPM package. You can install the package with the following command:

npm install @mediapipe/tasks-audio

If you want to import the task code through a content delivery network (CDN) service, load the task's Wasm files from the CDN as shown in Create the task.

Model

The MediaPipe Audio Transcriber task requires a speech recognition model that is compatible with this task. For more information on available models for Audio Transcriber, see the task overview Models section.

The task downloads the model from the URL that you pass in the options. The model is large, about 615 MB, so consider caching it in the browser, for example with the Cache Storage API, and passing the cached model contents to the task as baseOptions.modelAssetBuffer. The example code shows one way to do this.

Create the task

Use the createFromOptions() function to create the task. The createFromOptions() function accepts configuration options including the model, the delegate, the input sample rate, and whether to return interim results. For more information on configuration options, see Configuration options.

import {AudioTranscriber, FilesetResolver} from '@mediapipe/tasks-audio';

const audio = await FilesetResolver.forAudioTasks(
    'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-audio@latest/wasm');

const transcriber = await AudioTranscriber.createFromOptions(audio, {
  baseOptions: {
    modelAssetPath:
        'https://huggingface.co/litert-community/parakeet-tdt-0.6b-v3/resolve/main/parakeet_tdt_0.6b_v3_5s_i8_stateful.litertlm',
    delegate: 'CPU',
  },
  sampleRateHz: 16000,
  enableInterimResults: true,
  cacheDir: ':nocache',
});

If you don't need custom options, use AudioTranscriber.createFromModelPath(audio, modelUrl) instead.

Loading the model takes a moment. To get the first result faster, call prewarm() before you start transcribing:

await transcriber.prewarm();

To run the model on the GPU, set the delegate to 'GPU'. The GPU delegate uses WebGPU. If the browser doesn't support WebGPU, or the device has no suitable GPU, createFromOptions() rejects, and you can create the task with delegate: 'CPU' instead.

Loading the model and transcribing audio are compute-intensive and can block the page's main thread. To keep your user interface responsive, run the task in a web worker and send it audio from the main thread, as shown in the example code.

Configuration options

This task has the following configuration options for web applications:

Option Name Description Value Range Default Value
modelAssetPath URL of the speech recognition model file (.litertlm). Set this option in the base options. You can also pass the model contents as a Uint8Array or a ReadableStreamDefaultReader in baseOptions.modelAssetBuffer. URL None (required)
delegate Hardware used to run the model. Set this option in the base options. {"CPU", "GPU"} CPU
sampleRateHz Sample rate of the input audio, in Hz. If the input isn't 16000 Hz, the task resamples it to 16 kHz before transcribing it. Positive number 16000
enableInterimResults Whether to return interim segments while audio is streaming. Interim segments are early hypotheses that can change as more audio arrives. The task later returns a final segment for the same audio. Boolean false
modelArchitecture Model family. With the default value, the task detects the model family from the model file. Use the open-weights value for models such as Parakeet. {"AUTO", "OMNI"} "AUTO"
cacheDir Writable directory where the task caches compiled model artifacts, so that later starts are faster. On the web, set it to ":nocache", because the Wasm virtual file system doesn't persist across page loads. Directory path Not set

Prepare data

Audio Transcriber takes mono audio as a Float32Array of samples in the range [-1.0, 1.0]. If your audio isn't 16 kHz, set sampleRateHz to its sample rate and the task resamples it.

The following examples show how to prepare an audio clip and an audio stream with the Web Audio API:

Audio clip

// Decode an audio file (or any format the browser supports) and convert it
// to 16 kHz mono samples.
const encoded = await (await fetch('/path/to/audio.mp3')).arrayBuffer();
const decoded = await new AudioContext().decodeAudioData(encoded);
const offline = new OfflineAudioContext(
    1, Math.ceil(decoded.duration * 16000), 16000);
const source = offline.createBufferSource();
source.buffer = decoded;
source.connect(offline.destination);
source.start();
const samples = (await offline.startRendering()).getChannelData(0);

Audio stream

// Capture microphone audio at 16 kHz and receive it in chunks through an
// AudioWorklet.
const workletCode = `
  registerProcessor('capture', class extends AudioWorkletProcessor {
    process(inputs) {
      if (inputs[0][0]) this.port.postMessage(inputs[0][0].slice());
      return true;
    }
  });`;
const audioCtx = new AudioContext({sampleRate: 16000});
await audioCtx.audioWorklet.addModule(
    URL.createObjectURL(new Blob([workletCode], {type: 'text/javascript'})));
const stream = await navigator.mediaDevices.getUserMedia({audio: true});
const capture = new AudioWorkletNode(audioCtx, 'capture');
audioCtx.createMediaStreamSource(stream).connect(capture);
capture.port.onmessage = (event) => {
  // event.data is a Float32Array of 128 samples. Buffer it, for example
  // into 200 ms chunks, and transcribe each chunk.
  addToBuffer(event.data);
};

Run the task

Call transcribe() to transcribe an audio clip in a single call. For an audio stream, call transcribeStream() with each chunk of audio. When the audio ends, call finishStream() to process any remaining audio and get the final segments.

Audio clip

// Transcribe the whole clip in a single call.
const result = await transcriber.transcribe(samples);

Audio stream

// Transcribe live audio chunk by chunk.
async function onAudioChunk(chunk) {
  showResult(await transcriber.transcribeStream(chunk));
}

// Process the remaining audio and get the final segments.
async function onRecordingStopped() {
  showResult(await transcriber.finishStream());
}

Note the following:

  • Call the task's functions one at a time. Wait for each call to finish before you make the next one, for example by queuing the audio chunks.
  • When running on an audio stream, each transcribeStream() call returns the segments that became available for that chunk. A call can return no segments.
  • After a call to transcribe() or finishStream(), the next transcribeStream() call starts a new stream.

For a more complete example of running Audio Transcriber on an audio clip and an audio stream, see the example code.

Handle and display results

The Audio Transcriber returns an AudioTranscriberResult object for each call. The result contains a list of transcript segments. Each segment has its transcript, an isFinal flag that says whether it's final, and its fromTimestampMs and toTimestampMs timestamps. The following listing shows an example of the output data from this task:

AudioTranscriberResult:
  Segment #0:
    transcript: "Turn on the kitchen lights"
    is final: true
    from timestamp (milliseconds): 0
    to timestamp (milliseconds): 1840
  Segment #1:
    transcript: "and set a timer for"
    is final: false
    from timestamp (milliseconds): 1840
    to timestamp (milliseconds): 2600

Final segments don't change. Interim segments are returned only when enableInterimResults is set, and the task replaces them with final segments as more audio arrives. One way to show live captions is to append final text and redraw the interim text after it:

const finalText = [];

function showResult(result) {
  const interimText = [];
  for (const segment of result.segments) {
    if (segment.isFinal) {
      finalText.push(segment.transcript);
    } else {
      interimText.push(segment.transcript);
    }
  }
  captions.textContent = [...finalText, ...interimText].join(' ');
}

To get the combined text of all segments in a result, use result.text.

Release resources

When you no longer need the task, call close() to release the model and its memory:

transcriber.close();