The Web API of LiteRT-LM for **JavaScript and TypeScript** in the browser.

## Supported Models

The LiteRT-LM Web API supports the models in the
[Web LLM Models](https://huggingface.co/collections/litert-community/web-llm-models)
collection on Hugging Face, including Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B.
Load one by the URL of its model file, or by its repo with a `hf://` URI, which
chooses the repo's web model file (see [Load models from Hugging Face](https://developers.google.com/edge/litert-lm/js#hugging-face)):

    const engine = await Engine.create({
      // The model's Hugging Face repository
      model: 'hf://litert-community/gemma-4-E2B-it-litert-lm',
      // Or the URL of a model file, on Hugging Face or anywhere else
      // model: 'https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm/resolve/main/gemma-4-E2B-it-gpu.litertlm',
    });

Some of these models are gated: Hugging Face only lets users who have accepted
the model's license download them. To load one, pass the access token of such a
user (see [Gated models](https://developers.google.com/edge/litert-lm/js#gated-models)).

For vector embedding generation in the browser (such as EmbeddingGemma 2 using
`EmbeddingEngine`), see the [Embedding Models](https://developers.google.com/edge/litert-lm/embedding_models) guide.

## Introduction

Here is a sample REPL chat app built with the JavaScript API:

    <div id="out" style="white-space: pre-wrap; font-family: monospace;"></div>
    <input id="in" onkeydown="if(event.key === 'Enter') repl(this)">

    <script type="module">
      import { Engine } from 'https://cdn.jsdelivr.net/npm/@litert-lm/core/+esm';
      const engine = await Engine.create({
        // Load the Gemma 4 E2B model
        model: 'hf://litert-community/gemma-4-E2B-it-litert-lm',
        // Or use the E4B model by swapping in this line
        // model: 'hf://litert-community/gemma-4-E4B-it-litert-lm',
      });
      const chat = await engine.createConversation();

      window.repl = async (el) => {
        const text = el.value;
        el.value = ''; // Clear immediately
        out.append(`\n>>> ${text}\nAI: `);

        for await (const chunk of chat.sendMessageStreaming(text)) {
          out.append(chunk.content[0].text);
        }
      };
    </script>

> [!IMPORTANT]
> **Try our demo:** For a more complete chat experience, [try out our demo in Chrome](https://google-ai-edge.github.io/LiteRT-LM/web_demos/chat/index.html).

## Getting Started

LiteRT-LM is available as an npm package. You can install the latest version
from npm or directly import it from a CDN:

    # From npm
    npm i --save @litert-lm/core

    # From a CDN (in your JavaScript file)
    import * as litertlm from 'https://cdn.jsdelivr.net/npm/@litert-lm/core/+esm';

### Initialize the Engine

The `Engine` is the entry point to the API. It handles model loading, session
creation, and resource management. Remember to `delete` the engine to release
resources when the model is no longer needed.

**Note:** Initializing the engine can take several seconds to load the model.

    import {Engine, EngineSettings} from '@litert-lm/core';

    const engineSettings = {
      model: 'url/path/to/model.litertlm', // or a ReadableStream, or a Blob

      // You can configure context length and other settings here
      mainExecutorSettings: {
        maxNumTokens: 8192,
      },
    } satisfies EngineSettings;

    const engine = await Engine.create(engineSettings);

    // ... Use the engine to create a conversation ...

    // Delete the engine when done.
    await engine.delete();

#### Flexible Model Sources \& Streaming

The `model` field supports multiple input types:

- **URL string**: Fetches and streams weights directly over HTTP or HTTPS.
- **`hf://` URI** : Loads a model from the Hugging Face Hub (see [Load models from Hugging Face](https://developers.google.com/edge/litert-lm/js#hugging-face)).
- **`Blob` or `File`**: Loads models from local storage, IndexedDB, or the File System Access API.
- **`ReadableStream<Uint8Array>`**: Supports custom progressive streaming pipelines.

    // Example: Loading from a Blob or local File handle
    const fileInput = document.querySelector('input[type="file"]') as HTMLInputElement;
    const file = fileInput.files![0];

    const engine = await Engine.create({
      model: file,
    });

#### Load models from Hugging Face

A `hf://<org>/<repo>[@<revision>][/<path>]` URI loads a model from the Hugging
Face Hub. Without a path, LiteRT-LM chooses the model file in the repository for the
engine's backend. If several files match, it loads one and logs a warning that
lists the others, which you can load by giving their path:

    // The repo's web model file.
    model: 'hf://litert-community/gemma-4-E4B-it-litert-lm'

    // A specific file at a specific revision (branch, tag, or commit).
    model: 'hf://litert-community/gemma-4-E4B-it-litert-lm@main/gemma-4-E4B-it-gpu.litertlm'

The revision is resolved to its current commit before anything is downloaded,
so the file that's downloaded always matches its SHA-256.

#### Gated models

Hugging Face only lets users who have accepted a gated model's license download
it. To load a gated model, pass the access token of such a user, for example
from [Sign in with Hugging Face](https://huggingface.co/docs/hub/oauth), in a
`ModelSource`:

    const engine = await Engine.create({
      model: {
        url: 'hf://litert-community/Gemma3-1B-IT',
        accessToken: userAccessToken,
      },
    });

> [!CAUTION]
> **Caution:** Don't embed your own access token in a web page: everyone who loads the page can read it.

#### Custom WASM Binary Location

If you are self-hosting WASM files or deploying in offline environments, load
the WASM runtime before calling `Engine.create()`:

    import {loadLiteRtLm} from '@litert-lm/core';

    // Load from a custom directory containing the WASM binaries
    await loadLiteRtLm('/path/to/wasm/directory/');

### Create a Conversation

Once the engine is initialized, create a `Conversation` instance. You can
provide a `ConversationConfig` to customize system messages and sampling
parameters.

    import {ConversationConfig, SamplerType} from '@litert-lm/core';

    const config: ConversationConfig = {
      // Initial background messages and instructions
      preface: {
        messages: [
          {role: 'system', content: 'You are a helpful and concise coding assistant.'},
        ],
      },
      // Generation and sampling controls
      sessionConfig: {
        maxOutputTokens: 2048,
        samplerParams: {
          type: SamplerType.TOP_P,
          temperature: 0.7,
          p: 0.9,
          k: 40,
        },
      },
      // Optional optimizations
      enableConstrainedDecoding: true,
      prefillPrefaceOnInit: true,
      filterChannelContentFromKvCache: true,
    };

    const conversation = await engine.createConversation(config);

### Send Messages

You can send messages with or without streaming.

#### Non-Streaming Example

    // Simple string input
    let response = await conversation.sendMessage("What is the capital of France?");
    console.log(response.content[0].text);

    // Or with full message structure
    response = await conversation.sendMessage({role: 'user', content: '...'});

#### Streaming Example

    // sendMessageStreaming returns a ReadableStream of response chunks
    const stream = conversation.sendMessageStreaming('Tell me a long story.');

    for await (const chunk of stream) {
      // Chunks are Records containing pieces of the response
      for (const item of chunk.content) {
        if (item.type === 'text') {
          console.log(item.text);
        }
      }
    }

#### Cancel Generation

You can cancel an ongoing generation explicitly by calling `cancel()` on the
`Conversation` instance:

    // Cancel any ongoing generation
    conversation.cancel();

If you are streaming the response, exiting the `for await...of` loop early (such
as with `break`) will also automatically cancel the ongoing generation:

    for await (const chunk of stream) {
      if (shouldStop()) {
        break; // Cancels the stream and underlying generation
      }
    }

### Manage Conversation State

`Conversation` provides methods for branching, inspecting history, and
monitoring tokens:

    // 1. Fork a conversation into an independent branch without re-prefilling history
    const branchedChat = await conversation.clone();

    // 2. Inspect the current token count of the conversation context
    const tokenCount = await conversation.getTokenCount();
    console.log(`Tokens in context: ${tokenCount}`);

    // 3. Retrieve conversation history
    const history = await conversation.getHistory();
    console.log('History:', history);

    // 4. Retrieve inference performance benchmarks
    const benchmarkInfo = await conversation.getBenchmarkInfo();
    console.log(`Prefill rate: ${benchmarkInfo.lastPrefillTokensPerSecond} tokens/sec`);
    console.log(`Decode rate: ${benchmarkInfo.lastDecodeTokensPerSecond} tokens/sec`);

### Thinking

When running reasoning-capable models, the LiteRT-LM Web SDK allows generating
internal reasoning tokens ("thinking") before returning the final response.

You can toggle thinking mode by setting `enable_thinking` in `extra_context`
within the conversation preface:

    import { Engine } from '@litert-lm/core';

    // 1. Initialize the Engine
    const engine = await Engine.create({
      model: 'url/path/to/thinking_model.litertlm'
    });

    // 2. Configure thinking using extra_context in the conversation preface
    const conversation = await engine.createConversation({
      preface: {
        extra_context: {
          enable_thinking: true, // Set to false to disable thinking
        }
      },
      sessionConfig: {
        maxOutputTokens: 2048,
      }
    });

    // 3. Send a message
    const response = await conversation.sendMessage("Solve this logic puzzle step by step.");

    // 4. Access reasoning tokens and final answer
    if (response.channels?.['thought']) {
      console.log("Thought:", response.channels['thought']);
    }
    console.log("Answer:", response.content?.[0]?.text ?? response.content);

    // Clean up
    await engine.delete();

> [!NOTE]
> **Note:** The LiteRT-LM Web SDK supports toggling thinking mode using `extra_context` in the conversation preface. Dedicated token budget constraints are supported in the Python, Kotlin, and Swift SDKs.