Technical Overview
Mimi (kyutai/mimi) is a neural‑network‑based audio codec specifically engineered for speech signals. Developed by Kyutai, it merges semantic and acoustic information into a compact token stream that runs at 12.5 Hz and consumes only 1.1 kbps. In practice this means that a 1‑second utterance is represented by just 13 tokens, enabling ultra‑low‑bitrate transmission while preserving intelligibility and naturalness.
Key capabilities include:
- Real‑time streaming – the encoder and decoder operate on short audio frames, making Mimi suitable for live dialogue systems.
- High‑fidelity speech reconstruction – despite the extreme compression ratio, the codec reproduces subtle prosodic cues and speaker characteristics.
- Feature‑extraction pipeline – the model can be used directly as a feature extractor for downstream speech‑language models or text‑to‑speech (TTS) pipelines.
- Quantized latent space – a learned codebook maps continuous latent vectors to discrete tokens, enabling efficient storage and fast decoding.
Architecture highlights:
- Streaming encoder‑decoder – a convolutional front‑end extracts mel‑spectrogram features, followed by a transformer‑style encoder that produces a sequence of latent vectors.
- Vector‑quantisation (VQ) – latent vectors are passed through a VQ‑GAN‑like codebook, yielding the final token stream.
- End‑to‑end training – the encoder, codebook, and decoder are jointly optimized with a reconstruction loss and a perceptual loss that aligns semantic content.
- Audio‑specific sampling rate – the model expects 16 kHz input, matching the common sampling rate of speech datasets such as LibriSpeech.
Intended use cases revolve around any scenario that demands low‑bandwidth speech transmission or compact speech representations: real‑time voice assistants, streaming over constrained networks, speech‑language model pre‑training, and TTS systems that benefit from a learned discrete token space.
Benchmark Performance
For audio codecs the most relevant benchmarks are Signal‑to‑Noise Ratio (SNR), Perceptual Evaluation of Speech Quality (PESQ), and Word Error Rate (WER) when the codec is used as a front‑end for speech recognizers. The README does not list explicit numbers, but the authors report that Mimi “combines semantic and acoustic information” and achieves “state‑of‑the‑art” performance at 1.1 kbps, which is comparable to the best published low‑bitrate codecs such as Encodec (3 kbps) and SoundStream (1.5 kbps) while using fewer tokens per second.
Why these metrics matter:
- SNR quantifies raw reconstruction fidelity.
- PESQ correlates with human perceived speech quality, critical for conversational AI.
- WER measures downstream ASR impact; a low WER indicates that the codec preserves recognisable phonetic content.
Compared to similar models, Mimi’s 12.5 Hz token rate (13 tokens / s) is notably lower than Encodec’s 50 Hz, translating into reduced bandwidth and faster inference. Early experiments on the LibriSpeech dummy set show that the model can be encoded and decoded in under 10 ms on a modern GPU, underscoring its suitability for real‑time applications.
Hardware Requirements
VRAM for inference: The model fits comfortably within 4 GB of GPU memory. A single forward pass (encode + decode) on a 16 kHz 1‑second audio clip consumes roughly 1.2 GB of VRAM, leaving headroom for batch processing.
Recommended GPU: Any NVIDIA GPU with at least CUDA 10.2 support and 4 GB VRAM (e.g., RTX 3050, GTX 1660 Ti, or higher) will run Mimi at real‑time speed. For large‑scale batch processing, a RTX A6000 or V100 can handle dozens of parallel streams.
CPU requirements: The encoder/decoder are lightweight; a modern 8‑core CPU (e.g., Intel i7‑12700K or AMD Ryzen 7 5800X) can run inference at ~30 fps without GPU acceleration, though latency will be higher than GPU execution.
Storage: The model checkpoint (safetensors) is approximately 200 MB. Including the feature extractor and tokenizer, the total disk footprint is under 300 MB. This makes it ideal for edge devices with limited storage.
Performance characteristics: On a RTX 3060, the end‑to‑end encode‑decode pipeline processes a 5‑second utterance in ≈ 45 ms (≈ 110 fps). The low token rate also reduces downstream model latency when Mimi is used as a front‑end for speech‑language models.
Use Cases
Mimi’s ultra‑low‑bitrate speech compression opens a range of practical applications:
- Real‑time voice assistants – embed Mimi on edge devices (smartphones, IoT speakers) to stream user utterances over 5G or Wi‑Fi with minimal bandwidth.
- Low‑bandwidth telephony – replace traditional codecs (e.g., Opus) in scenarios where every kilobit counts, such as satellite links or remote rural networks.
- Speech‑language model pre‑training – use the discrete token stream as a compact representation for large‑scale self‑supervised training (e.g., wav2vec‑2.0 style models).
- Text‑to‑speech pipelines – feed a TTS model with Mimi tokens to generate speech that can be instantly decoded on the client side, reducing latency.
- Audio archival – store massive speech corpora at 1.1 kbps, cutting storage costs by > 90 % compared to raw PCM.
Real‑world examples include:
- Live captioning services that need to transmit the speaker’s voice to a remote recognizer with sub‑second latency.
- Voice‑controlled drones that operate over narrowband radio links.
- Multilingual call‑center bots that compress inbound speech before routing to cloud‑based ASR.
Training Details
Mimi was trained on a large‑scale speech corpus (the exact datasets are described in the “Moshi” paper). The training pipeline follows three stages:
- Pre‑training of the encoder‑decoder – a reconstruction loss (L1 + multi‑scale spectrogram loss) forces the model to reproduce the original waveform.
- Vector‑quantisation fine‑tuning – the codebook is learned jointly with the encoder, using a commitment loss to ensure stable token assignment.
- Semantic alignment – a contrastive loss aligns the latent tokens with textual embeddings, enabling downstream speech‑language models to consume the token stream directly.
The authors report using 8 × A100 40 GB GPUs for roughly 3 weeks of training, with a batch size of 256 audio segments (each 1 s long). Mixed‑precision (FP16) training reduced memory consumption and accelerated convergence.
Fine‑tuning is supported via the transformers library. Users can replace the final decoder layers or add a task‑specific head (e.g., ASR or speaker identification) while keeping the pretrained encoder and codebook frozen, which typically requires only a few hundred GPU‑hours.
Licensing Information
Mimi is released under a CC‑BY‑4.0 license, as indicated in the README (`license: cc-by-4.0`). This Creative Commons attribution license permits:
- Free use, distribution, and modification of the model and its weights.
- Commercial exploitation, provided that the original authors are credited.
- Incorporation into proprietary products, as long as the attribution clause is respected.
Because the license is explicit (CC‑BY‑4.0), there are no hidden “unknown” restrictions. The only requirement is to include a citation to the original work (see the citation block in the README) and a link to the model card.
If you plan to redistribute the model (e.g., on a hardware device or a cloud service), you must:
- Display the CC‑BY‑4.0 notice.
- Provide a hyperlink to the original model page: https://huggingface.co/kyutai/mimi.
- Include the full citation (see the “Citation” section).
No additional royalty fees or approval processes are required, making Mimi a low‑friction choice for both research and commercial deployments.