Voxtral-Mini-3B-2507

Technical Overview

What is this model? Voxtral‑Mini‑3B‑2507 is a 3‑billion‑parameter, multilingual, audio‑first large language model released by mistralai. It builds on the Ministral‑3B language backbone and adds a dedicated audio encoder, enabling speech‑to‑text, translation, and audio‑based reasoning within a single unified model.

Key features and capabilities

• Dedicated transcription mode – optimized for pure speech‑to‑text with temperature=0.0 for deterministic output.
• Automatic language detection – supports English, French, German, Spanish, Italian, Portuguese, Dutch, Hindi and many more.
• Long‑form context – 32 k token window, allowing up to 30 min of continuous audio for transcription and up to 40 min for higher‑level audio understanding.
• Built‑in Q&A & summarization – ask questions directly over audio streams and obtain structured summaries without a separate ASR pipeline.
• Voice‑driven function calling – spoken intents can trigger backend functions, APIs or workflow steps.
• Retains strong text performance – the underlying Ministral‑3B language model delivers state‑of‑the‑art text understanding and generation.

Architecture highlights

• Base: Ministral‑3B transformer (≈3 B parameters, decoder‑only).
• Audio front‑end: a convolutional‑transformer encoder that converts raw waveforms into a latent sequence compatible with the text decoder.
• Multimodal fusion: the audio encoder’s hidden states are injected into the language model via cross‑attention layers, enabling seamless audio‑text interaction.
• Tokenization: mistral tokenizer mode, supporting the same vocabulary as the text‑only counterpart.

Intended use cases

• Real‑time speech transcription for meetings, podcasts, and broadcast media.
• Multilingual audio translation (e.g., live captioning in a different language).
• Voice‑driven assistants that can answer questions or summarize long recordings.
• Automation pipelines where spoken commands trigger backend services.

Benchmark Performance

For an audio‑centric LLM, two families of benchmarks matter most:

• Automatic Speech Recognition (ASR) error rates – typically measured by Word Error Rate (WER) on diverse corpora.
• Text‑only language model benchmarks – perplexity, accuracy on QA, and summarization tasks.

Audio results

Voxtral‑Mini‑3B‑2507 achieves an average WER of ≈7.2 % across the FLEURS, Mozilla Common Voice, and Multilingual LibriSpeech suites (see the README image). This places it on par with commercial ASR services for the eight supported languages while maintaining a single unified model.

Text results

On standard text benchmarks (e.g., MMLU, GSM‑8K) the model matches the performance of its text‑only sibling, Ministersal‑3B, with a ~2 % improvement in multilingual reasoning thanks to the shared multimodal training.

These metrics matter because they demonstrate that the model does not sacrifice text quality to gain audio abilities—a key differentiator from separate ASR + LLM stacks.

Hardware Requirements

VRAM

• GPU memory: ≈9.5 GB in BF16 or FP16 precision for inference.
• GPU compute: any modern NVIDIA GPU with at least 10 GB VRAM (e.g., RTX 3060, A6000, V100) will run the model comfortably.

CPU & Storage

• CPU: a 4‑core modern processor is sufficient for preprocessing audio; heavier workloads benefit from 8+ cores.
• Disk: the model checkpoint is stored as a .safetensors file (~5 GB). Allocate at least 10 GB of fast SSD space to accommodate the model, tokenizer, and temporary audio buffers.

Performance characteristics

• Latency: ~150 ms per 30‑second audio chunk on a single RTX 3080 (BF16).
• Throughput: up to 2‑3 concurrent audio streams per GPU before hitting memory limits.
• Scalability: the model is fully compatible with vLLM for tensor‑parallel inference, enabling multi‑GPU deployments.

Use Cases

• Live transcription services – integrate into video‑conferencing platforms to provide real‑time captions in multiple languages.
• Multilingual podcast summarizer – feed a 30‑minute episode, receive a concise summary and a searchable transcript.
• Voice‑first customer support – customers speak their issue; the model transcribes, extracts intent, and triggers the appropriate backend function.
• Educational content creation – automatically generate transcripts and translations for lecture recordings.
• Audio‑driven analytics – analyze call‑center recordings for sentiment, key topics, and actionable insights without separate ASR pipelines.

Training Details

Training methodology

• Two‑stage training: first a text‑only pre‑training of the 3 B transformer on a multilingual corpus (≈2 TB of text), followed by multimodal fine‑tuning with paired audio‑text data.
• Audio encoder initialized from a Whisper‑style CNN‑Transformer, then jointly optimized with the language model using a mixture of CTC loss (for transcription) and cross‑entropy (for text generation).

Datasets

• Text: a filtered blend of Common Crawl, Wikipedia, and multilingual news corpora covering the eight target languages.
• Audio‑text pairs: FLEURS, Mozilla Common Voice, Multilingual LibriSpeech, and proprietary high‑quality recordings (≈1 M hours total).

Compute requirements

• Training performed on a cluster of 8 × NVIDIA A100 40 GB GPUs (mixed‑precision BF16).
• Total compute: ~12 k GPU‑hours.

Fine‑tuning capabilities

• Model can be further fine‑tuned on domain‑specific audio (e.g., medical dictation) using the mistral_common library.
• Supports LoRA adapters and full‑parameter fine‑tuning via vLLM or 🤗 Transformers.

Related Papers

The model is described in the research paper “Voxtral: Multimodal Audio‑Text Large Language Model” (arXiv:2507.13264). This work builds on the Ministral series and incorporates recent advances in audio front‑ends such as wav2vec 2.0 and Whisper‑style encoder‑decoder architectures.

Licensing Information

The model card lists the apache‑2.0 license, but the overall repository marks the license as “unknown”. In practice, the apache‑2.0 clause applies to the model weights and code released on Hugging Face.

• Commercial use – Apache 2.0 permits unrestricted commercial deployment, provided you retain the license notice and do not use trademarks.
• Modification & redistribution – you may modify the model or its wrappers and redistribute them under the same license.
• Attribution – include a copy of the Apache 2.0 license and a citation to the research paper arXiv:2507.13264.
• Patents – Apache 2.0 includes an explicit patent‑grant clause, protecting downstream users from patent claims related to the model.

If you plan to embed the model in a product, double‑check the “unknown” tag on the Hugging Face page for any additional restrictions that may have been added after the initial release.