Audio transcription guide for iOS

The MediaPipe Audio Transcriber task lets you transcribe speech in audio data into text on the device. You can use this task to transcribe a live microphone stream or an audio clip. These instructions show you how to use the Audio Transcriber with iOS apps.

For more information about the capabilities, models, and configuration options of this task, see the Overview.

Code example

The MediaPipe Tasks example code is a basic implementation of an Audio Transcriber app for iOS. The example uses the microphone on an iOS device to transcribe speech continuously. It shows interim results while you speak and final transcript segments with timestamps.

You can use the app as a starting point for your own iOS app, or refer to it when modifying an existing app. You can refer to the Audio Transcriber example code on GitHub.

Download the code

The following instructions show you how to create a local copy of the example code using the git command line tool.

To download the example code:

Clone the git repository using the following command:

git clone https://github.com/google-ai-edge/mediapipe-samples

Optionally, configure your git instance to use sparse checkout, so you have only the files for the Audio Transcriber example app:
```
cd mediapipe-samples
git sparse-checkout init --cone
git sparse-checkout set examples/audio_transcriber/ios
```

After creating a local version of the example code, add the model to the app as described in the example's README. Then open the project in Xcode and run the app. For instructions, see the Setup guide for iOS.

Key components

The following files contain the main code for the Audio Transcriber example app:

AudioTranscriberViewModel.swift: Creates the Audio Transcriber, streams microphone audio to it, and collects the interim and final transcript segments.
AudioCaptureEngine.swift: Captures microphone audio with AVAudioEngine and converts it to 48 kHz mono samples.
AudioTranscriberView.swift: Implements the SwiftUI user interface.

Setup

This section describes key steps for setting up your development environment and code projects to use Audio Transcriber. For general information on setting up your development environment for using MediaPipe tasks, including platform version requirements, see the Setup guide for iOS.

Dependencies

Audio Transcriber uses the MediaPipeTasksAudio library, which can be installed using Swift Package Manager (SPM). The library is compatible with both Swift and Objective-C apps and does not require any additional language-specific setup.

To add the MediaPipeTasksAudio library to your Xcode project:

With your project open in Xcode, select File > Add Package Dependencies....
In the search field in the upper right, enter the repository URL: https://github.com/google-ai-edge/mediapipe
Select the mediapipe package and choose your dependency rule.
Click Add Package, then select MediaPipeTasksAudio and add it to your application target.

To transcribe the microphone, add the NSMicrophoneUsageDescription key to your app's Info.plist file and configure an AVAudioSession that allows recording.

Model

The MediaPipe Audio Transcriber task requires a speech recognition model that is compatible with this task. For more information about the available models for Audio Transcriber, see the task overview Models section.

The model is too large to bundle in your app. Download the model the first time your app needs it, store it in your app's Application Support directory, and pass the absolute path of the file to the task. During development, you can add the model file to your Xcode project instead. For instructions, refer to Managing files and folders in your Xcode project.

Use the BaseOptions.modelAssetPath property to specify the path to the model. For a code example, see the next section.

Create the task

You can create the Audio Transcriber task by calling one of its initializers. The AudioTranscriber(options:) initializer sets values for the configuration options. If you don't need custom options, you can use the AudioTranscriber(modelPath:) initializer to create an Audio Transcriber with the default options. For more information about configuration options, see Configuration options.

Swift

import MediaPipeTasksAudio

let options = AudioTranscriberOptions()
options.baseOptions.modelAssetPath = modelURL.path
options.sampleRateHz = 16000
options.enableInterimResults = true

let audioTranscriber = try AudioTranscriber(options: options)

Objective-C

@import MediaPipeTasksAudio;

MPPAudioTranscriberOptions *options = [[MPPAudioTranscriberOptions alloc] init];
options.baseOptions.modelAssetPath = modelPath;
options.sampleRateHz = 16000;
options.enableInterimResults = YES;

NSError *error = nil;
MPPAudioTranscriber *audioTranscriber =
    [[MPPAudioTranscriber alloc] initWithOptions:options error:&error];

Loading the model takes a moment. To get the first result faster, call prewarm() on a background queue before the user starts speaking.

Configuration options

This task has the following configuration options for iOS apps:

Option Name	Description	Value Range	Default Value
`modelAssetPath`	Absolute path to the speech recognition model file (`.litertlm`) on the device. Set this option in the base options. In-memory model buffers aren't supported.	File path	None (required)
`delegate`	Hardware used to run the model. Set this option in the base options.	{`.CPU`, `.GPU`}	CPU
`sampleRateHz`	Sample rate of the input audio, in Hz. If the input isn't 16000 Hz, the task resamples it to 16 kHz before transcribing it.	Positive number	`16000`
`enableInterimResults`	Whether to return interim segments while audio is streaming. Interim segments are early hypotheses that can change as more audio arrives. The task later returns a final segment for the same audio.	Boolean	`NO`
`modelArchitecture`	Model family. With the default value, the task detects the model family from the model file. Use the open-weights value for models such as Parakeet.	{`.auto`, `.omni`}	`.auto`
`cacheDir`	Writable directory where the task caches compiled model artifacts, so that later starts are faster. To turn caching off, set it to `":nocache"`.	Directory path	Not set

Prepare data

Audio Transcriber takes mono audio as float samples in the range [-1.0, 1.0]. If your audio isn't 16 kHz, set sampleRateHz to its sample rate when you create the task. For example, the microphone input of an iPhone is usually 48 kHz.

Audio stream

import AVFoundation

let audioEngine = AVAudioEngine()
let inputNode = audioEngine.inputNode
let format = inputNode.outputFormat(forBus: 0)
// Create the task with options.sampleRateHz = format.sampleRate.

inputNode.installTap(onBus: 0, bufferSize: 4800, format: format) { buffer, _ in
  guard let channel = buffer.floatChannelData?[0] else { return }
  // Copy the first channel, and transcribe it off the audio thread.
  let samples = Array(UnsafeBufferPointer(start: channel, count: Int(buffer.frameLength)))
  transcriptionQueue.async { transcribe(samples) }
}
try audioEngine.start()

Audio clip

import AVFoundation

let file = try AVAudioFile(
  forReading: audioURL, commonFormat: .pcmFormatFloat32, interleaved: false)
let buffer = AVAudioPCMBuffer(
  pcmFormat: file.processingFormat, frameCapacity: AVAudioFrameCount(file.length))!
try file.read(into: buffer)
// Create the task with options.sampleRateHz = file.processingFormat.sampleRate.

Run the task

Call transcribe(floatBuffer:length:endOfStream:) with each chunk of audio. When the audio ends, call finishStream() to process any remaining audio and get the final segments. For a clip, pass all of the samples in one call, and then call finishStream().

Swift

func transcribe(_ samples: [Float]) {
  let result = samples.withUnsafeBufferPointer { pointer in
    try? audioTranscriber.transcribe(
      floatBuffer: pointer.baseAddress, length: UInt(pointer.count), endOfStream: false)
  }
  if let result { showResult(result) }
}

// When the user stops speaking:
transcriptionQueue.async {
  if let finalResult = try? audioTranscriber.finishStream() {
    showResult(finalResult)
  }
}

Objective-C

NSError *error = nil;
MPPAudioTranscriberResult *result =
    [audioTranscriber transcribeFloatBuffer:samples
                                     length:sampleCount
                                endOfStream:NO
                                      error:&error];

// When the user stops speaking:
MPPAudioTranscriberResult *finalResult =
    [audioTranscriber finishStreamWithError:&error];

Note the following:

The transcribe, finishStream(), and prewarm() methods block the current thread until they finish. Call them on a serial background queue, never on the main thread or the audio tap.
Each transcribe call returns the segments that became available for that chunk. A call can return no segments.
You can also pass samples as an array of numbers with transcribe(samples:endOfStream:). The float buffer method avoids an extra copy and is better for live audio.

Handle and display results

The Audio Transcriber returns an AudioTranscriberResult object for each call. The result contains a list of transcript segments. Each segment has its text, a flag that says whether it's final, and its start and end timestamps. The following listing shows an example of the output data from this task:

AudioTranscriberResult:
  Segment #0:
    transcript: "Turn on the kitchen lights"
    is final: true
    from timestamp (milliseconds): 0
    to timestamp (milliseconds): 1840
  Segment #1:
    transcript: "and set a timer for"
    is final: false
    from timestamp (milliseconds): 1840
    to timestamp (milliseconds): 2600

Final segments don't change. Interim segments are returned only when enableInterimResults is set, and the task replaces them with final segments as more audio arrives. One way to show live captions is to append final text and redraw the interim text after it:

Swift

var finalText = ""

func showResult(_ result: AudioTranscriberResult) {
  var interimText = ""
  for segment in result.segments {
    if segment.isFinal {
      finalText += segment.transcript + " "
    } else {
      interimText += segment.transcript + " "
    }
  }
  let caption = finalText + interimText
  DispatchQueue.main.async { captionLabel.text = caption }
}

Objective-C

NSMutableString *interimText = [NSMutableString string];
for (MPPAudioTranscriberSegment *segment in result.segments) {
  if (segment.isFinal) {
    [self.finalText appendFormat:@"%@ ", segment.transcript];
  } else {
    [interimText appendFormat:@"%@ ", segment.transcript];
  }
}
NSString *caption = [self.finalText stringByAppendingString:interimText];
dispatch_async(dispatch_get_main_queue(), ^{
  self.captionLabel.text = caption;
});

To get the combined text of all segments in a result, use the text property.

Release resources

When you no longer need the task, call close() to release the model and its memory.

Swift

try audioTranscriber.close()

Objective-C

[audioTranscriber closeWithError:&error];

Audio transcription guide for iOS Stay organized with collections Save and categorize content based on your preferences.

Code example

Download the code

Key components

Setup

Dependencies

Model

Create the task

Swift

Objective-C

Configuration options

Prepare data

Audio stream

Audio clip

Run the task

Swift

Objective-C

Handle and display results

Swift

Objective-C

Release resources

Swift

Objective-C

Audio transcription guide for iOS