TalTechNLP/Voxtral-Mini-4B-Realtime-estonian-2609

Voxtral Mini 4B Realtime Estonian 2609

TalTechNLP/Voxtral-Mini-4B-Realtime-estonian-2609 is an Estonian-focused version of Mistral AI's Voxtral Mini 4B Realtime 2602. It is a causal streaming automatic speech recognition (ASR) model intended for live subtitling, meeting transcription, voice interfaces, and other applications in which partial transcripts must be emitted while audio is still arriving.

The original approximately 4-billion-parameter architecture is unchanged. It contains an approximately 3.4B-parameter language model and a causal approximately 970M-parameter audio encoder. This release was fully fine-tuned for Estonian: the audio encoder, multimodal projector, and language model were all updated.

The repository contains weights in both supported layouts:

• model.safetensors is the Transformers checkpoint.
• consolidated.safetensors is the Mistral/vLLM checkpoint. Its audio query and key projection rows use the interleaved RoPE layout expected by stock vLLM.

Weights are stored in BF16. The model expects mono, 16 kHz audio; callers should downmix and resample input when necessary.

Usage

vLLM (recommended for realtime serving)

Install a vLLM version with Voxtral Realtime support and its audio dependencies, then start the OpenAI-compatible server:

vllm serve TalTechNLP/Voxtral-Mini-4B-Realtime-estonian-2609 \
  --compilation-config '{"cudagraph_mode":"PIECEWISE"}'

The server exposes the /v1/realtime WebSocket endpoint. See the vLLM realtime audio client example for a complete client. Use deterministic decoding (temperature=0). The transcription delay is configured in tekken.json; 480 ms is a good starting point for balancing latency and accuracy. Longer sessions require a correspondingly large --max-model-len.

Bundled directory transcription script

This repository includes transcribe_voxtral_realtime_vllm.py, a standalone command-line utility for transcribing a directory of audio files through vLLM's Python API. It drives the realtime interface directly and does not start an HTTP/WebSocket server. vllm_voxtral_realtime_compat.py is its bundled support module and must remain beside the script.

For example, the following command transcribes every WAV file in audio/, writes the corresponding text files to transcripts/, and prints transcript deltas while decoding:

python transcribe_voxtral_realtime_vllm.py \
  --model-id TalTechNLP/Voxtral-Mini-4B-Realtime-estonian-2609 \
  --input-dir audio \
  --output-dir transcripts \
  --input-pattern '*.wav' \
  --output-pattern '*.txt' \
  --live-output true \
  --dtype bfloat16

The script loads common audio formats through mistral-common, downmixes them to mono, resamples them to 16 kHz, and feeds them to vLLM in incremental chunks. Existing outputs are skipped by default; pass --overwrite true to replace them. Standard five-column word-timing files can be generated alongside the transcripts:

python transcribe_voxtral_realtime_vllm.py \
  --model-id TalTechNLP/Voxtral-Mini-4B-Realtime-estonian-2609 \
  --input-dir audio \
  --output-dir transcripts \
  --ctm-output-pattern '*.ctm'

Use --tensor-parallel-size, --gpu-memory-utilization, and --max-model-len to adjust deployment to the available GPUs and expected recording length. Run python transcribe_voxtral_realtime_vllm.py --help for all options. The frontend-alignment and tracing arguments are diagnostic features; their defaults should be retained for normal transcription.

Transformers

Voxtral Realtime is supported in Transformers 5.2 and newer. The following example transcribes a local audio file:

from mistral_common.tokens.tokenizers.audio import Audio
from transformers import AutoProcessor, VoxtralRealtimeForConditionalGeneration

model_id = "TalTechNLP/Voxtral-Mini-4B-Realtime-estonian-2609"

processor = AutoProcessor.from_pretrained(model_id)
model = VoxtralRealtimeForConditionalGeneration.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
)

audio = Audio.from_file("audio.wav", strict=False)
audio.resample(processor.feature_extractor.sampling_rate)
inputs = processor(audio.audio_array, return_tensors="pt")
inputs = inputs.to(model.device, dtype=model.dtype)

output_ids = model.generate(**inputs, do_sample=False)
transcript = processor.batch_decode(output_ids, skip_special_tokens=True)[0]
print(transcript)

For genuine incremental streaming, use vLLM's realtime API or implement the chunked feedback loop described by the base model card.

Training data

Training data was assembled from four sources. Durations below are the sums of the training segments and are not necessarily the durations of unique source recordings. In particular, the manually transcribed subset contains the original audio and 0.9x and 1.1x speed-perturbed copies. Its unaugmented portion contains 200,155 segments (1,378.1 hours).

Source	Candidate segments / hours	After CTC filtering	Used by the trainer
TalTech Estonian Speech Dataset 1.0 (including speed perturbation)	604,917 / 4,161.5 h	590,363 / 3,980.0 h	581,557 / 3,896.3 h
Semisupervised Estonian television (ETV) speech with automatically generated transcripts (using TalTechNLP/whisper-large-v3-turbo-et-verbatim)	573,675 / 3,458.2 h	570,520 / 3,441.4 h	567,813 / 3,418.7 h
ERR web-video speech with context-biased automatically generated subtitles	337,314 / 2,197.0 h	333,156 / 2,171.4 h	333,150 / 2,171.3 h
Total	1,515,906 / 9.816 h	1,494,040 / 9,592.7 h	1,482,521 / 9,486.3 h

Transcript filtering

Candidate transcripts were independently decoded with TalTechNLP/xls-r-300m-et. For comparison, references and hypotheses were lowercased, all Unicode punctuation was removed, and whitespace was normalized. An utterance was kept when its word error rate was at most 0.5.

Of the 1,572,075 candidate segments:

• 70,060 were rejected because their utterance-level WER exceeded 0.5;
• 7,965 exceeded the CTC decoder's 40-second duration limit; and
• 10 had an empty reference after normalization.

The corpus WER of all successfully decoded, non-empty candidates before thresholding was 15.00%. The filter retained 1,494,040 segments from 56,067 recordings. The training loader then applied a stricter duration range of 0.2--30.0 seconds, excluding another 11,519 segments and leaving 1,482,521 training examples.

Streaming targets

Word timings were produced by CTC forced alignment with mms-meta/mms-zeroshot-300m. The alignment text was romanized with uroman using the Estonian language code; the output targets retained their original spelling and trailing punctuation. Words were placed into the streaming target sequence according to their aligned end times, with streaming padding tokens inserted between emission points.

Fine-tuning details

The released model is checkpoint 16,000 from a three-epoch training run. At this checkpoint the trainer had completed 2.7625 epochs.

Hyperparameter	Value
Training precision	BF16
Distributed training	4 GPUs, one process per GPU
Per-device batch size	8
Gradient accumulation	8 steps
Effective global batch size	256 examples
Optimizer	Fused AdamW
Learning rate	2e-5
Schedule	Linear decay
Warmup	100 steps
Weight decay	0.0
Adam betas / epsilon	0.9, 0.999 / 1e-8
Maximum gradient norm	1.0
Loss	Causal LM cross-entropy + 1e-4 z-loss
Trainable parameters	Full model (audio encoder, projector, and language model)
LoRA	Not used
Gradient checkpointing	Enabled
Sampling	Shuffled duration-grouped batches
Padding multiple	8 tokens
Random seed	0

Each batch used one randomly sampled transcription delay. The possible delays were every 80 ms from 80 through 2,400 ms, inclusive. This exposes the model to different latency/accuracy tradeoffs during fine-tuning rather than optimizing it for only one delay.

Evaluation

At step 16,000, the model obtained the following internal validation result:

Dataset	WER
Kõnetõlke benchmark	6.8%

Intended use and limitations

This model is intended primarily for Estonian realtime speech transcription. Likely applications include live captions, meeting and media transcription, and voice interfaces. Although the base model is multilingual, non-Estonian quality was not preserved or evaluated after fine-tuning.

Users should expect errors with noisy or far-field audio, overlapping speech, strong accents or dialects, code-switching, rare names, numbers, and domains that are poorly represented in the training material. The model does not perform speaker diarization and may omit, repeat, or hallucinate words. Its output should be reviewed before use in legal, medical, safety-critical, or other high-stakes settings.

Much of the training corpus has automatically generated transcripts. The CTC filter reduces gross disagreement but cannot establish that a transcript is correct, and it may favor examples that resemble the filtering model's own predictions. The data is also dominated by broadcast and web media and should not be assumed to represent all Estonian speakers or acoustic conditions.

License and attribution

The model weights are released under the Apache License 2.0. They are derived from mistralai/Voxtral-Mini-4B-Realtime-2602, which is also licensed under Apache-2.0.

The model license does not replace any rights or conditions associated with input recordings or transcripts. Users are responsible for ensuring that their use of the model and any audio they process respects applicable privacy, copyright, data-protection, and other legal requirements.

For architecture and base-model details, cite the Voxtral Realtime technical report and acknowledge Mistral AI's base model.