WASM-backed text chunker supporting sentence-aware character and word chunking.
Signature:
export declare class DefaultTextChunker implements TextChunker
Methods
| Method | Modifiers | Description |
|---|---|---|
| chunk(text) | Splits the given text into overlapping chunks according to the configured chunk size and overlap. | |
| create(wasmFileset, options) | static |
Initializes the Wasm runtime and creates a new DefaultTextChunker instance. |
DefaultTextChunker.chunk()
Splits the given text into overlapping chunks according to the configured chunk size and overlap.
Signature:
chunk(text: string): Promise<string[]>;
Parameters
| Parameter | Type | Description |
|---|---|---|
| text | string | Input document text to split. |
Returns:
Promise<string[]>
DefaultTextChunker.create()
Initializes the Wasm runtime and creates a new DefaultTextChunker instance.
Signature:
static create(wasmFileset: WasmFileset, options: { maxChunkSize: number; chunkOverlap: number; mode?: ChunkingMode; respectSentences?: boolean; }): Promise<DefaultTextChunker>;
Parameters
| Parameter | Type | Description |
|---|---|---|
| wasmFileset | WasmFileset | A configuration object that provides the location of the Wasm binary and its loader. |
| options | { maxChunkSize: number; chunkOverlap: number; mode?: ChunkingMode; respectSentences?: boolean; } | Chunking configuration options. |
Returns:
Promise<DefaultTextChunker>