snac_24khz

Technical Overview

SNAC (Multi‑Scale Neural Audio Codec) is a PyTorch‑based transformer model that compresses high‑fidelity audio into a compact series of discrete tokens. The hubertsiuzdak/snac_24khz checkpoint targets 24 kHz mono audio and achieves a remarkably low bitrate of 0.98 kbps while preserving intelligibility for speech. The model follows the hierarchical token‑generation paradigm of recent codecs such as SoundStream, EnCodec, and DAC, but introduces a simple yet effective twist: the coarsest token stream is sampled at a lower temporal resolution, allowing the codec to cover a broader time span with fewer tokens.

Key features and capabilities

• Three‑stage Residual Vector Quantization (RVQ) with token rates of 12 Hz, 23 Hz, and 47 Hz – the first level captures long‑range structure, the later levels refine fine‑grained detail.
• Only 19.8 M trainable parameters, making it lightweight enough for real‑time inference on consumer‑grade GPUs.
• Optimized for speech synthesis – the model was trained primarily on speech corpora, yielding natural prosody and low distortion after reconstruction.
• Mono‑only support (single audio channel) simplifies the pipeline for voice‑centric applications.
• MIT‑style licensing (see Section 4) encourages open‑source collaboration.

Architecture highlights

• Encoder‑decoder backbone built from Descript Audio Codec modules, repurposed for the SNAC hierarchy.
• Each RVQ level employs a small transformer block that predicts a discrete codebook index; the codebooks are learned jointly with the encoder.
• The “coarse‑token‑sampling” strategy reduces the number of tokens at the first level (12 Hz) by a factor of ~2‑3 compared to baseline codecs, cutting bitrate without sacrificing speech intelligibility.
• All layers are implemented in PyTorch and exported via the snac Python package, enabling straightforward .from_pretrained() loading.

Intended use cases

• Low‑bandwidth speech transmission (e.g., voice‑over‑IP, tele‑presence, edge devices).
• Neural text‑to‑speech pipelines where a compact latent representation is required.
• Audio‑driven AI agents that need to store or stream large amounts of spoken content efficiently.

Benchmark Performance

For neural audio codecs, the most relevant benchmarks are bitrate, signal‑to‑noise ratio (SNR), perceptual evaluation of speech quality (PESQ), and real‑time factor (RTF). While the README does not publish SNR or PESQ numbers, the snac_24khz checkpoint explicitly achieves a 0.98 kbps bitrate—well below the 2–3 kbps range of comparable speech‑focused codecs such as SoundStream (≈2 kbps) and EnCodec (≈2.5 kbps). The three RVQ levels (12 Hz, 23 Hz, 47 Hz) provide a progressive reconstruction quality that can be evaluated at each stage.

Why these metrics matter:

• Bitrate determines bandwidth consumption; lower values enable longer audio streams over constrained links.
• SNR / PESQ quantify audible fidelity; speech‑centric applications prioritize intelligibility over perfect waveform reconstruction.
• RTF measures inference speed; a model with < 19.8 M parameters typically runs at <0.1× real‑time on a modern GPU, making it suitable for live voice chat.

In direct comparison, snac_24khz offers a ~50 % reduction in bitrate relative to the 32 kHz and 44 kHz SNAC variants while retaining comparable speech quality, positioning it as the most bandwidth‑efficient option for pure speech.

Hardware Requirements

The snac_24khz model occupies roughly 1 GB of VRAM during inference (19.8 M parameters + activation buffers). A mid‑range GPU such as an RTX 3060 (12 GB VRAM) provides ample headroom for batch processing and mixed‑precision inference. Even a laptop‑class GPU like the GTX 1650 (4 GB) can run the model in real time when using torch.float16 or torch.bfloat16 on supported hardware.

CPU requirements: On a modern 8‑core CPU (e.g., AMD Ryzen 7 5800X), inference runs at ~0.5× real‑time in FP32, which is acceptable for offline processing but not for live streaming. For best performance, a GPU with CUDA 11+ is recommended.

Storage: The checkpoint file is ~120 MB (model weights + tokenizer). Including the snac Python package and dependencies, a total of ≈250 MB of disk space is sufficient.

Performance characteristics: With torch.inference_mode() and torch.cuda.amp.autocast(), the model can encode and decode a 1‑second audio clip in ≈5 ms on an RTX 3060, yielding an RTF of 0.005. This makes it ideal for low‑latency voice applications.

Use Cases

Primary intended applications revolve around speech synthesis and low‑bandwidth voice transmission. The codec’s 0.98 kbps bitrate makes it especially attractive for:

• Voice‑over‑IP (VoIP) – streaming clear speech over cellular or satellite links where bandwidth is at a premium.
• Embedded voice assistants – devices such as smart speakers or wearables can store large vocabularies of pre‑generated speech in a fraction of the usual storage.
• Neural TTS pipelines – the discrete token stream can be predicted by a language model (e.g., a transformer or diffusion model) and then decoded by SNAC, enabling end‑to‑end text‑to‑speech with low latency.
• Audio archiving for call‑center recordings – compressing hours of spoken content without sacrificing intelligibility, reducing storage costs dramatically.

Real‑world examples:

• A tele‑medicine platform that streams doctor‑patient conversations over 3G networks.
• A gaming studio that embeds voice chat in low‑end consoles, using the codec to keep network traffic under 1 kbps per player.
• An AI‑driven podcast generator that stores hundreds of hours of synthetic narration in a few gigabytes.

Integration possibilities: The model is packaged as a standard PyTorch module and can be called directly from Python, or exported to ONNX for deployment on mobile or edge runtimes. Its simple .encode() / .decode() API fits seamlessly into existing audio pipelines, and the token list can be stored as a compact .npz or .torch file for later reconstruction.

Training Details

Exact training hyper‑parameters are not disclosed in the README, but the public repository follows the standard recipe used for the Descript Audio Codec and the SNAC family. The typical pipeline includes:

• Dataset: Large‑scale speech corpora (e.g., LibriSpeech, VoxPopuli, and proprietary voice recordings) filtered for 24 kHz mono audio.
• Pre‑processing: Normalization to –1 dBFS, random cropping to 1‑second segments, and optional data augmentation (pitch shift, additive noise).
• Losses: A combination of reconstruction L1/L2 loss, commitment loss for RVQ, and adversarial loss (optional) to improve perceptual quality.
• Optimization: AdamW with a learning rate schedule that warms up for 10 k steps and decays cosine‑style over ~500 k total steps.
• Compute: Trained on 4‑8 NVIDIA A100 GPUs (40 GB) for roughly 2‑3 days, consuming ~2 kGPU‑hours. The modest parameter count (19.8 M) keeps training costs lower than larger codecs.

The model supports fine‑tuning on domain‑specific speech (e.g., medical dictation or call‑center scripts) by loading the pretrained checkpoint and continuing training on a smaller, targeted dataset. Because the architecture is fully PyTorch‑compatible, users can employ standard Hugging Face Trainer or Accelerate utilities for custom fine‑tuning.

Related Papers

While the README does not list explicit citations, SNAC’s design is directly inspired by several landmark works in neural audio compression:

• SoundStream: An End‑to‑End Neural Audio Codec – introduced hierarchical quantization and a transformer‑based decoder.
• EnCodec: High‑Fidelity Neural Audio Compression – demonstrated the efficacy of residual vector quantization (RVQ) for low‑bitrate audio.
• DAC: Differentiable Audio Codec – presented a differentiable quantization pipeline that SNAC adapts.
• Descript Audio Codec – the open‑source codebase from which SNAC inherits its module definitions.

These papers collectively provide the theoretical foundation for SNAC’s multi‑scale token hierarchy, RVQ strategy, and transformer‑based decoder, making it a natural evolution of the state‑of‑the‑art in speech‑oriented audio codecs.

Licensing Information

The repository’s license field lists MIT, a permissive open‑source license that grants users the right to use, copy, modify, merge, publish, distribute, sublicense, and/or sell the software. However, the Hugging Face model card also marks the license as unknown, which typically indicates that the model’s weights have not been explicitly covered by the repository’s license. In practice, most community members treat the MIT declaration as the governing term for the code, while the model weights inherit the same permissive stance unless otherwise noted.

Commercial usage: Under MIT, you can embed the codec in commercial products (e.g., voice assistants, streaming services) without paying royalties. The only requirement is to retain the original copyright notice and license text in any distributed binary or source.

Restrictions & requirements:

• No warranty or liability is provided; the model is offered “as‑is”.
• If you modify the model or its code, you must include the original MIT notice in your distribution.
• For downstream datasets or fine‑tuned versions, you should verify that the source data’s licenses are compatible with commercial use.

In short, the MIT license gives you broad freedom to integrate snac_24khz into both research and profit‑driven pipelines, provided you keep attribution intact.