Matrix execution engine

Coral NPU's Zvt matrix execution engine implements the RISC-V ISA extensions for high-throughput matrix multiplication computation. These instructions are described in the RISC-V specification document Zvt Family of Vector Matrix Extensions (v0.4). The official v1.0 release of this specification is scheduled for late 2026. This functionality is also referred to as the Vector Matrix Extensions (VME).

The ISA extensions provide specialized configuration registers and instructions for high-throughput tensor manipulation, matrix multiplication, and efficient data orchestration.

The VME-Zvt matrix extensions build on the Zve32x RVV vector extensions. The 128-bit vector registers v0 – v31 serve as a data source for matrix operands, and the existing vector instructions are used for pre- and post-processing of matrices.

Note: The definitive reference for the matrix instructions is the official RISC-V specification – always refer to that document for complete information. This page gives an overview and summary of the matrix instructions.

In Coral NPU, the matrix execution engine accelerates compute-dense ML workloads:

• Outer-product multiply-accumulator (MAC): A Multiply-Accumulate (MAC) operation multiplies two numbers together and adds the result to an existing value stored in an accumulator.
• Transformer prefill: The prefill phase in transformer models is the initial step of LLM inference in which the model processes the entire input prompt in parallel to build its foundational internal state and compute the Key-Value (KV) cache.
• Speculative decoding: Speculative decoding is an inference-time optimization for large language models (LLMs) that generates multiple tokens per decoding step instead of one. This technique speeds up AI text generation by using a small, fast model to guess multiple words ahead and a large model to check all the guesses at once.
• Enables Gemma 3 270M end-to-end: Gemma is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models. Gemma models are provided with open weights, allowing you to tune and deploy them in your applications.

The Coral MLIR compiler explicitly prepares the matrix math for the Coral NPU hardware, slicing massive data blocks to fit perfectly into the 16 internal matrix tile accumulators.

RISC-V VME-Zvt extensions

The Vector Matrix Extensions (VME) are a matrix compute extension for AI/ML applications based on an outer-product formulation of matrix-multiply. A basic VME matmul operation takes two vector registers, computes the outer product of the operands, and accumulates the results into a large accumulator. This targets efficient scaling from small- to high-performance implementations with high power efficiency and architectural simplicity.

VME is one of several approaches to matrix extensions for RISC-V, alongside IME (Integrated Matrix Extension). IME operates solely out of the existing RVV vector register file, so its performance for embedded and application-class processors is limited by vector register file capacity. VME instead remains closely coupled to a hart and its RVV implementation, reading vector registers for source operands and leveraging RVV for surrounding AI/ML computation, while adding an accumulator register file into which matmul results accumulate.

The VME extension has the following advantages:

• A relatively simple matrix extension based on outer-product matrix multiplication, with no new 2D load/store instructions, operating on matrix memory data in standard row- or column-major order
• Leverages the existing RVV engine of Coral NPU and the AI/ML vector extensions.
• Instructions for matmul compute and for moving data into and out of accumulator registers, plus vsetvl-like instructions to set matrix parameters for subsequent matmul instructions.
• Matrix-dimension-agnostic operation, similar to RVV being vector-length-agnostic and using similar vsetvl and vl mechanisms.

VME-Zvt micro-architecture in Coral NPU

Coral NPU's matrix engine is the core computational module for high-throughput matrix multiplication and tensor operations. It includes a dedicated array of processing elements (PEs) and accumulators.

The Coral NPU matrix engine operates as a decoupled co-processor to the vector engine, receiving micro-ops (µops) and data through dedicated command and register file interfaces. It uses a multi-stage decoupled pipeline to maximize throughput:

• Command dequeue: Commands are received from the vector dispatcher and staged in a handshake-based command queue.
• Data orchestration: Synchronizes with the Vector Register File (VRF) and accumulators, fetching source operands (va, vb) and managing subtile access.
• PE array computation: The core PE array performs fused multiply-add (FMA) operations. The pipeline includes internal logic to handle bubbles (REMVPIPEBUBB) and optimize multiplier/adder utilization.
• Accumulator management: Results are written back to dedicated high-speed accumulators or local memory, supporting saturating arithmetic and floating-point exception tracking.

Edge cases

Coral NPU handles the following edge cases:

• Precise exceptions: A fault during matrix accumulation is trapped and reported at the end of the command sequence.
• Partial matrix operations: Operations with dimensions not perfectly mapping to PE array width are handled through operand padding and masked writes.
• Accumulator saturation: Saturating arithmetic triggers when values exceed the defined precision, setting the appropriate status flags.

Matrix instructions

The definitive reference for these instructions is the official specification document Zvt Family of Vector Matrix Extensions (v0.4). A summary of the instructions is given here for quick reference.

Software must execute mset* configuration instructions to establish the M x K x N tiling dimensions, before issuing systolic matrix compute micro-ops.

Other instructions are provided to move rows and columns of the matrix arrays — moving them either to or from the vector registers or memory.

Instruction	Description
Matrix configuration
msetmtype rs1,rs2	Write the mtype and vtype CSRs to the values in the rs1 and rs2 registers.
msetmtypei	Similar to msetmtype except: 1) the new mtype CSR value is supplied by a zero-extended immediate, 2) the vtype.vsew field value is supplied by a zero-extended immediate, and 3) all other vtype fields are zeroed.
msettn rd,rs1	Set tn/vl to the value in rs1, constrained by the current matrix configuration; sets matrix dimension N. Operates similarly to vsetvl, updating vl from rs1 and writing the new vl to rd. (See note below.*)
msettm rd,rs1	Set tm to the value in rs1, constrained by the current matrix configuration – sets tm to be the lesser of rs1, LMULxEVE, and ETE. Sets matrix dimension M. (See note below.*)
msettk rd,rs1	Set tk to the value in rs1, constrained by the current matrix configuration – sets tk to the lesser of rs1 and KMAX. Sets matrix dimension K. (See note below.*)
Load/store tile subset to memory
vtle* rs2,rs1	Load tile subset from memory, with rs1 = memory address, rs2 = TSS. For example: vtle32 rs2,rs1
vtse* rs2,rs1	Store tile subset to memory, with rs1 = memory address, rs2 = TSS. For example: vtse32 rs2,rs1
Move tile subset between tile state and vector registers
vtmv.v.t vd,rs1	Move tile subset to vector register group, with vd = destination vector register and rs1 = TSS of source.
vtmv.t.v rs1,vs2	Move vector register group to tile subset, with vs2 = source vector register and rs1 = TSS of destination.
Matrix arithmetic
vtfmm.tvv mtd,vs2,vs1	Floating-point matrix multiply; use the values of tm, tn, and tk to determine what computations to perform. vs2 specifies A vector register group, vs1 specifies B vector register group, mtd is C destination tile specifier. **
vtfmm.alt.tvv mtd,vs2,vs1	Floating-point matrix multiply; use the values of tm, tn, and tk to determine what computations to perform. vs2 specifies A vector register group, vs1 specifies B vector register group, mtd is C destination tile specifier. **
vtmmu.tvv mtd,vs2,vs1	Integer matrix multiply; use the values of tm, tn, and tk to determine what computations to perform. vs2 specifies A vector register group, vs1 specifies B vector register group, mtd is C destination tile specifier. **
vtmms.tvv mtd,vs2,vs1	Integer matrix multiply; use the values of tm, tn, and tk to determine what computations to perform. vs2 specifies A vector register group, vs1 specifies B vector register group, mtd is C destination tile specifier. **
Miscellaneous
vtzero mtd	Write 0 to each element of the tm by tn submatrix of the destination matrix tile.
vtdiscard	Inform the runtime that the matrix tile state is no longer useful and need not be saved.

* For msettm and msettk, the new field value is written back to the destination register rd. The msettn instruction uses the standard vsetvl datapath to write the updated vl to rd.

** vtype holds SEW and altfmt; mtype holds TWIDEN

• data type of vs2 is supplied by opcode,SEW
• data type of vs1 is supplied by altfmt,SEW
• data type of mtd is supplied by opcode,SEW,TWIDEN

---