Getting started with LiteRT

LiteRT is Google's on-device inference engine for deploying ML and GenAI models on edge platforms, with tools to convert, optimize, and run them.

The latest LiteRT 2.x release introduces the CompiledModel API, a modern runtime interface designed to maximize hardware acceleration. While the Interpreter API (formerly TensorFlow Lite) remains available for backward compatibility, the CompiledModel API is the recommended choice for developers seeking state-of-the-art performance in on-device AI applications.

1 Prerequisites 2 Install packages 3 Obtain model 4 Run & verify 5 Accelerate
Docs: Android Web LiteRT-LM Migration

Development workflow

LiteRT runs inference entirely on-device on Android, iOS, Web, IoT, and on desktop/laptop. On every platform, the workflow follows the same five steps, with links to detailed instructions.

1. Prerequisites

Before developing with LiteRT, verify that your workstation environment meets the following minimum toolchain and SDK requirements:

Platform / Toolchain Minimum Requirements Recommended Tooling Target Accelerators
Python Python 3.9–3.12 pip 23.0+ or uv CPU, GPU, Qualcomm NPU
Android (Kotlin / C++) Android Studio Ladybug+ (2024.2.1+)
Min SDK 24+ (Android 7.0+)		 Android NDK r26a+ (for C++) CPU, GPU (OpenCL/OpenGL), NPU
Web (JavaScript / TS) Modern browser with WebGPU support
(Chrome 113+, Edge 113+, Safari 17.4+)		 Node.js 18+ (bundler / npm) WebGPU, WASM (CPU), WebNN (NPU)
C++ (Native & Embedded) CMake 3.19+
C++17 compiler (GCC 10+, Clang 12+, MSVC 2019+)		 Ninja build system CPU (XNNPACK), GPU (OpenCL/Vulkan), NPU
iOS (Swift) Xcode 15+
Deployment target iOS 15.0+ / macOS 12.0+		 CocoaPods 1.12+ or Swift Package Manager CPU, GPU (Metal), Core ML

2. Install LiteRT packages

Install LiteRT SDK packages across your target platforms:

Python

# Platform: Linux, macOS, Windows (Python 3.9+)
# Core LiteRT (Vision, Audio, CompiledModel)
pip install ai-edge-litert

# LiteRT-LM (On-device LLMs & Generative AI)
pip install litert-lm-api

Command-line tools install separately — see the CLI Tools tab.

Android (Kotlin)

// Platform: Android (API 24+) | In app/build.gradle.kts
dependencies {
    // Core LiteRT runtime with CompiledModel API.
    // GPU and NPU acceleration are built in; no extra artifact
    // is needed. Select one with Accelerator in Options (Step 5).
    implementation("com.google.ai.edge.litert:litert:2.2.0")

    // LiteRT-LM runtime for on-device LLMs & Gemma
    implementation("com.google.ai.edge.litertlm:litertlm-android:latest.release")
}

iOS (Swift)

// Platform: iOS 15+, macOS 12+

// LiteRT-LM ships a versioned Swift package. In Package.swift:
.package(url: "https://github.com/google-ai-edge/LiteRT-LM", from: "0.16.0")

// In target dependencies:
.product(name: "LiteRTLM", package: "LiteRT-LM")

// LiteRT Runtime iOS distribution is still pre-release:
// the LiteRTSwift CocoaPod publishes nightly builds only, and
// Swift Package Manager support is in progress.
pod 'LiteRTSwift'   // nightly builds

Web (JavaScript)

# Platform: Web (Browsers & Node.js)
# Core LiteRT.js runtime for vision & audio models
npm install @litertjs/core

# LiteRT-LM for on-device LLM token streaming
npm install @litert-lm/core

C++

# Platform: Linux, Android NDK, macOS, Windows
# In CMakeLists.txt (CMake 3.19+)
# Link prebuilt LiteRT C++ SDK (see /edge/litert/next/cpp_sdk)
target_link_libraries(${PROJECT_NAME} PRIVATE
    litert_cc_api
)

CLI Tools

# Platform: Linux, macOS, Windows
# LiteRT Model Tools CLI (conversion, quantization, benchmarking)
pip install litert-cli-nightly

# LiteRT-LM CLI (on-device LLM prompt testing & serving)
pip install litert-lm

3. Obtain a model

A LiteRT model is represented in an efficient portable format known as FlatBuffers, which uses the .tflite file extension.

Quickstart: Download a sample model

To test the code examples in Step 4, download a pre-trained MobileNetV2 image classification model (.tflite):

curl -L -o mobilenet_v2.tflite \
  https://storage.googleapis.com/tfweb/litertjs_demo_models/mobilenetv2/torchvision_mobilenet_v2.tflite

This provides a ready-to-run model (ImageNet 1,000-class classification, input shape 1x3x224x224 float32) for immediate verification.

4. Run inference and verify results

LiteRT lets you run ML models entirely on-device with high performance across Android, iOS, Web, desktop, and IoT platforms.

The following tabbed examples demonstrate the complete lifecycle—loading the mobilenet_v2.tflite model downloaded in Step 3, allocating buffers, running inference, and inspecting output classifications:

Python

# Platform: Linux, macOS, Windows • Accelerators: CPU / GPU / NPU
import numpy as np
from ai_edge_litert.compiled_model import CompiledModel, HardwareAccelerator

# 1. Load model and compile for target accelerator (CPU / GPU / NPU)
model = CompiledModel.from_file("mobilenet_v2.tflite", HardwareAccelerator.CPU)

# 2. Preallocate input and output memory buffers
input_buffers = model.create_input_buffers(0)
output_buffers = model.create_output_buffers(0)

# 3. Populate input buffer with test tensor (1x3x224x224 float32 for MobileNetV2)
input_data = np.zeros((1, 3, 224, 224), dtype=np.float32)
input_buffers[0].write(input_data)

# 4. Execute on-device inference
model.run_by_index(0, input_buffers, output_buffers)

# 5. Read output tensor and inspect classification logits (1,000 ImageNet classes)
output = output_buffers[0].read(1000, np.float32)
top_class = np.argmax(output)
print("LiteRT Python inference completed successfully!")
print(f"Output shape: {output.shape}")
print(f"Top predicted class index: {top_class}, score: {output[top_class]:.4f}")

Android (Kotlin)

// Platform: Android (API 24+) • Accelerators: Qualcomm / MediaTek NPU, GPU, CPU
import android.util.Log
import com.google.ai.edge.litert.Accelerator
import com.google.ai.edge.litert.CompiledModel

// 1. Initialize and compile model (bundle mobilenet_v2.tflite into app/src/main/assets/)
// For files outside assets, use CompiledModel.create(file.absolutePath, options)
CompiledModel.create(
    context.assets,
    "mobilenet_v2.tflite",
    // Target accelerator: CPU, GPU, or NPU
    CompiledModel.Options(Accelerator.CPU)
).use { compiledModel ->
    // 2. Preallocate input buffer with test image tensor (1x3x224x224 float32)
    val inputBuffers = compiledModel.createInputBuffers()
    val inputData = FloatArray(1 * 3 * 224 * 224) { 0.0f }
    inputBuffers[0].writeFloat(inputData)

    // 3. Run inference (automatically creates and returns output buffers)
    val outputBuffers = compiledModel.run(inputBuffers)

    // 4. Read and verify classification logits (1,000 ImageNet classes)
    val output = outputBuffers[0].readFloat()
    val topClass = output.indices.maxByOrNull { output[it] } ?: -1
    Log.d("LiteRT", "Inference completed successfully!")
    Log.d("LiteRT", "Output logits size: ${output.size} classes")
    Log.d("LiteRT", "Top predicted class: $topClass (score: ${output[topClass]})")
}

iOS (Swift)

// Platform: iOS 15+, macOS 12+ • Accelerators: Apple Silicon GPU / Neural Engine / CPU
import Foundation
import LiteRT

// 1. Initialize environment (defaults to CPU acceleration)
let env = try Environment()

// 2. Compile model from app bundle (add mobilenet_v2.tflite to Xcode bundle resources)
guard let modelPath = Bundle.main.path(forResource: "mobilenet_v2", ofType: "tflite") else {
  fatalError("Model file not found in bundle")
}
let compiledModel = try CompiledModel(filePath: modelPath, environment: env)

// 3. Allocate and populate input buffers (1x3x224x224 float32)
let inputBuffers = try compiledModel.createInputBuffers()
let outputBuffers = try compiledModel.createOutputBuffers()
let inputData = [Float](repeating: 0.0, count: 1 * 3 * 224 * 224)
try inputBuffers[0].write(inputData)

// 4. Execute inference
try compiledModel.run(inputs: inputBuffers, outputs: outputBuffers)

// 5. Read and inspect output classification results (1,000 classes)
let output: [Float] = try outputBuffers[0].read()
print("LiteRT Swift inference completed successfully!")
print("Output logits count: \(output.count) classes")
if let maxIdx = output.indices.max(by: { output[$0] < output[$1] }) {
  print("Top predicted class: \(maxIdx) (score: \(output[maxIdx]))")
}

Web (JavaScript)

// Platform: Web (Browsers & Node.js) • Accelerators: WebGPU / WebAssembly
import {loadLiteRt, loadAndCompile, Tensor} from '@litertjs/core';

// 1. Initialize LiteRT WebAssembly runtime
await loadLiteRt('https://cdn.jsdelivr.net/npm/@litertjs/core/wasm/');

// 2. Load and compile public model targeting WebGPU acceleration
// (WebGPU requires HTTPS / secure context; fallback to CPU if unavailable)
const model = await loadAndCompile(
  'https://storage.googleapis.com/tfweb/litertjs_demo_models/mobilenetv2/torchvision_mobilenet_v2.tflite',
  { accelerator: 'webgpu' }
);

// 3. Prepare input tensor (1x3x224x224 float32)
const inputData = new Float32Array(1 * 3 * 224 * 224);
const inputTensor = new Tensor(inputData, [1, 3, 224, 224]);

// 4. Run inference with input tensors
const outputs = await model.run([inputTensor]);

// 5. Read 1,000-class output logits and inspect results
const output = await outputs[0].data();
let topClass = 0;
for (let i = 1; i < output.length; i++) {
  if (output[i] > output[topClass]) topClass = i;
}
console.log('LiteRT.js inference completed successfully!');
console.log('Output logits length:', output.length);
console.log(`Top predicted class: ${topClass} (score: ${output[topClass].toFixed(4)})`);

// 6. Clean up model and tensor resources to prevent WebGPU/WASM memory leaks
inputTensor.delete();
outputs.forEach(tensor => tensor.delete());
model.delete();

C++

// Platform: Linux, Android NDK, macOS, Windows • Accelerators: CPU / GPU / NPU
#include <algorithm>
#include <iostream>
#include <vector>
#include "absl/types/span.h"
#include "litert/cc/litert_compiled_model.h"
#include "litert/cc/litert_environment.h"
#include "litert/cc/litert_macros.h"

litert::Expected<void> RunInference() {
  // 1. Initialize LiteRT runtime environment
  LITERT_ASSIGN_OR_RETURN(auto env, litert::Environment::Create({}));

  // 2. Load model and compile for CPU acceleration
  LITERT_ASSIGN_OR_RETURN(
      auto compiled_model,
      litert::CompiledModel::Create(env, "mobilenet_v2.tflite",
                                    litert::HwAccelerators::kCpu));

  // 3. Preallocate input and output buffers
  LITERT_ASSIGN_OR_RETURN(auto input_buffers,
                          compiled_model.CreateInputBuffers());
  LITERT_ASSIGN_OR_RETURN(auto output_buffers,
                          compiled_model.CreateOutputBuffers());

  // 4. Fill input buffer with test image tensor (1x3x224x224 float32)
  std::vector<float> input_data(1 * 3 * 224 * 224, 0.0f);
  LITERT_RETURN_IF_ERROR(
      input_buffers[0].Write<float>(absl::MakeConstSpan(input_data)));

  // 5. Invoke model inference
  LITERT_RETURN_IF_ERROR(
      compiled_model.Run(input_buffers, output_buffers));

  // 6. Read and verify classification output logits (1,000 classes)
  std::vector<float> output_data(1000);
  LITERT_RETURN_IF_ERROR(
      output_buffers[0].Read<float>(absl::MakeSpan(output_data)));

  auto max_it = std::max_element(output_data.begin(), output_data.end());
  int top_class = std::distance(output_data.begin(), max_it);
  std::cout << "LiteRT C++ inference completed successfully!" << std::endl;
  std::cout << "Output logits size: " << output_data.size() << " classes" << std::endl;
  std::cout << "Top predicted class: " << top_class << " (score: " << *max_it << ")" << std::endl;

  return {};
}

int main() {
  if (auto status = RunInference(); !status) {
    std::cerr << "Inference failed: " << status.Error() << std::endl;
    return 1;
  }
  return 0;
}

5. Choose a hardware backend

The Step 4 examples all target the CPU. To accelerate the same model, change only the accelerator you pass when compiling — HardwareAccelerator.GPU in Python, Accelerator.GPU in Kotlin, Options.setHardwareAccelerators([.gpu]) in Swift, litert::HwAccelerators::kGpu in C++, or {accelerator: 'webgpu'} on the Web. Nothing else changes: the CompiledModel API handles delegate setup and falls back automatically when an op is unsupported. See the on-device inference guide for more details.

Android iOS / macOS Web Linux Windows IoT
CPU XNNPACK XNNPACK XNNPACK XNNPACK XNNPACK XNNPACK
GPU OpenGL
OpenCL		 Metal
WebGPU		 WebGPU WebGPU
OpenCL		 WebGPU
OpenCL		 WebGPU
NPU Samsung Exynos
Google Tensor
MediaTek
Qualcomm		 - - Qualcomm Intel
Qualcomm		 Qualcomm

Use your own model

Once the quickstart runs, swap in a model of your own. Start from a pre-trained catalog, or convert a model you already have:

Build a reference app

Explore end-to-end reference applications built with the CompiledModel API across vision, audio, and generative AI:

Modality Task Hardware Acceleration Sample Implementation
Vision Image Segmentation CPU, GPU, NPU (JIT & AOT) • Kotlin Android App (CameraX)
• C++ App (Async Execution)
Audio Speech Recognition (ASR) CPU, NPU • Kotlin Android App (Real-time Mic Stream)
GenAI Semantic Similarity CPU, GPU, NPU • EmbeddingGemma C++ Demo

Advanced: author custom models and ops

When pre-trained or converted models do not meet your performance or architecture requirements, the Generative API (litert-torch) provides PyTorch building blocks for composing or fine-tuning Transformer models (such as Gemma and TinyLlama) with mobile-friendly KV-caching and built-in support for .tflite conversion. See the Generative API Guide.

Additional documentation and support