Learn how to configure global defaults and per-model settings for the LiteRT-LM CLI using configuration files.

## Overview

The LiteRT-LM CLI provides a centralized configuration system (`config.json`) to manage model execution parameters, hardware backend selection, sampling options, and performance optimizations across commands.

### Why Use Configuration Files?

Configuration files solve two key operational challenges:

1. **Eliminate Repetitive Flags** : Instead of passing long command-line flags (such as `--backend=gpu --cpu-thread-count=16 --max-num-tokens=2048`) every time you execute `litert-lm run` or `litert-lm benchmark`, you can define persistent global or per-model defaults.
2. **Configure OpenAI-Compatible Server** : When running the [OpenAI-Compatible Server](https://developers.google.com/edge/litert-lm/cli/openai_server) (`litert-lm serve`), standard OpenAI API requests (e.g., `POST /v1/chat/completions`) do not include LiteRT-LM-specific parameters like hardware backend selection (`npu` vs `gpu`), vision/audio backends, or Multi-Token Prediction (MTP / speculative decoding). `config.json` allows the server to automatically resolve these settings based on the requested `model` identifier.

### Configuration File Location

By default, the CLI looks for a JSON configuration file at:

### Linux/macOS

`~/.litert-lm/config.json`

### Windows

`%USERPROFILE%\.litert-lm\config.json`

You can override the configuration file path with the `--config /path/to/config.json` to any CLI command to specify an explicit configuration file.

## Configuration Structure

The configuration file (`config.json`) is a JSON object with two primary top-level sections:

    {
      "default": {
        /* Global ModelConfig settings applied to all models */
      },
      "models": {
        "model-id-1": {
          /* ModelConfig overrides for model-id-1 */
        },
        "model-id-2": {
          /* ModelConfig overrides for model-id-2 */
        }
      }
    }

- **`default`** : A `ModelConfig` object containing default settings applied to any model executed by the CLI.
- **`models`** : A map of model IDs to `ModelConfig` objects. Settings specified for a model override any matching fields in `default`.

## Settings Precedence

When resolving execution parameters for a model, LiteRT-LM applies settings according to a strict hierarchy defined in `config.json` (see [Configuration Structure](https://developers.google.com/edge/litert-lm/cli/configuration#configuration-structure)) and runtime command-line arguments. Higher-priority sources override lower-priority ones:

| Precedence Rank | Source | Example |
|---|---|---|
| **1 (Highest)** | Runtime Arguments | `litert-lm run --backend=gpu` |
| **2** | Model-Specific Config (`models.<model_id>`) | `"models": { "gemma4-e2b": { "backend": "gpu" } }` |
| **3** | Global Default Config (`default`) | `"default": { "backend": "gpu" }` |
| **4** | Model Metadata / Chat Template | Embedded default sampling parameters |
| **5 (Lowest)** | Engine Built-in Defaults | Engine default backend (`cpu`) |

> [!NOTE]
> **Note:** For details on model IDs and model registry management, see [Model Management](https://developers.google.com/edge/litert-lm/cli/model_management).

## OpenAI-Compatible Server Integration

When running the [OpenAI-Compatible Server](https://developers.google.com/edge/litert-lm/cli/openai_server) (`litert-lm serve`), the server dynamically resolves configuration parameters for incoming HTTP requests:

1. A client sends an OpenAI-compatible request, for example:

       curl http://localhost:9379/v1/chat/completions \
         -H "Content-Type: application/json" \
         -d '{
           "model": "gemma4-e2b",
           "messages": [{"role": "user", "content": "Hello!"}]
         }'

2. The server extracts the `"model"` identifier (`"gemma4-e2b"`).

3. The server checks `config.json` for a matching key under `models.gemma4-e2b`. If found, it merges those settings over the global `default` configuration.

4. The underlying engine is initialized or reused with the resolved hardware acceleration (`gpu`/`npu`), thread counts, and speculative decoding options.

This enables you to define custom server aliases with tailored performance profiles (such as lightweight CPU models vs high-performance GPU/MTP models) without modifying client code.

## Supported Configuration Keys

All properties in a `ModelConfig` object are optional. The following table summarizes all supported configuration keys, their data types, constraints, supported CLI versions, and descriptions.

> [!CAUTION]
> **Caution:** Hardware backend settings (such as `backend`, `audio_backend`, or `vision_backend`) and model feature flags depend on model and engine compatibility. While you can request a specific execution backend in `config.json`, not all models support every backend or feature. If a requested backend is incompatible with the model, initialization may fail. It is recommended to rely on the model's embedded default configuration unless explicitly testing alternative execution backends.

| Key | Data Type | Allowed Values / Constraints | Description | Added in Version |
|---|---|---|---|---|
| `backend` | String | `"cpu"`, `"gpu"`, `"npu"` | Execution backend for main LLM token generation. | `0.15.0` |
| `cpu_thread_count` | Integer | Min: `1` | Number of CPU threads to allocate when running on the CPU backend. | `0.15.0` |
| `audio_backend` | String | `"cpu"`, `"gpu"`, `"npu"` | Execution backend for audio preprocessing and audio feature extraction. | `0.15.0` |
| `vision_backend` | String | `"cpu"`, `"gpu"`, `"npu"` | Execution backend for image/vision model encoder processing. | `0.15.0` |
| `cache` | String | `"disk"`, `"memory"`, `"no"` | Key-value and compilation caching strategy. | `0.15.0` |
| `max_num_tokens` | Integer | Min: `1` | Maximum token limit combining input prompt context and output generation window. | `0.15.0` |
| `temperature` | Number | Min: `0.0` | Sampling temperature for text generation. Higher values increase randomness. | `0.15.0` |
| `top_p` | Number | `0.0` to `1.0` | Nucleus sampling probability cutoff threshold. | `0.15.0` |
| `top_k` | Integer | Min: `1` | Top-k sampling candidate limit. | `0.15.0` |
| `seed` | Integer | Any integer | Seed value for deterministic output generation. | `0.15.0` |
| `gpu_decode_steps_per_sync` | Integer | Min: `1` | Number of GPU decode steps executed between host synchronization points. | `0.15.0` |
| `speculative_decoding` | Boolean | `true`, `false` | Enables speculative decoding / Multi-Token Prediction (MTP) for supported models. | `0.15.0` |
| `thinking` | Boolean | `true`, `false` | Enables reasoning/thinking mode for supporting thinking models. | `0.15.0` |
| `thinking_budget` | Integer | Min: `-1` | Token budget limit for model reasoning steps (`-1` for unlimited/default). | `0.15.0` |
| `activation_data_type` | String | `"fp32"`, `"fp16"`, `"int16"`, `"int8"` | Overrides the activation data type precision used during model execution. | `0.15.0` |

## Example Configuration

Here is a `config.json` example demonstrating global defaults and per-model overrides:

    {
      "default": {
        "backend": "gpu",       // use GPU as the main backend, by default
        "cpu_thread_count": 16, // when using CPU, use 16 threads, by default
      },
      "models": {
        "gemma4-e2b": {
          "speculative_decoding": true,  // enable MTP
        },
        "gemma4-26b": {
          "max_num_tokens": 65536,   // set larger context for agentic use case
          "thinking": true,          // enable thinking by default
          "thinking_budget": 4096    // set a thinking budget
        }
      }
    }