Technical Overview
EnCodec is a state‑of‑the‑art neural audio codec released by Meta AI. The model facebook/encodec_24khz operates at a 24 kHz sampling rate and is designed for real‑time compression and decompression of speech, music, and general audio. It transforms raw waveforms into a compact quantized latent representation and reconstructs high‑fidelity audio on‑the‑fly.
Key features and capabilities
- Streaming encoder‑decoder architecture that supports both chunked (non‑streamable) and continuous (streamable) inference.
- Quantized latent space with lightweight Transformer heads for aggressive bitrate reduction while preserving perceptual quality.
- Multiscale spectrogram adversarial loss that minimizes artifacts across a wide frequency range.
- Loss‑balancer mechanism that decouples hyper‑parameter scaling from the raw loss magnitude, stabilising training.
- Feature‑extraction pipeline tag, enabling seamless integration with downstream audio tasks (e.g., TTS, speech enhancement).
Architecture highlights
- Convolutional front‑end with weight‑normalisation for the streamable variant.
- Hierarchical encoder that progressively downsamples the waveform to a latent sequence.
- Vector‑quantiser (VQ) that discretises the latent into
audio_codesandaudio_scales. - Two lightweight Transformer modules that further compress the VQ output.
- Decoder mirrors the encoder, reconstructing the waveform from the quantised representation.
Intended use cases
- Real‑time audio streaming where bandwidth is limited (e.g., gaming voice chat, live broadcasting).
- Pre‑processing step for speech‑to‑text, text‑to‑speech, or music generation pipelines.
- Audio archival where high fidelity is required but storage must be minimised.
Benchmark Performance
Benchmarks for neural audio codecs focus on perceptual quality (e.g., MOS, PESQ), bitrate, and latency. The EnCodec paper (arXiv:2210.13438) reports a Mean Opinion Score (MOS) of 4.3 at a 6 kbps bitrate for 24 kHz speech, outperforming Opus (4.0 MOS) at comparable bitrate. For music, EnCodec achieves ~4.0 MOS at 8 kbps, with a latency of < 10 ms on a single RTX 3090 GPU, satisfying real‑time constraints.
Compared to other neural codecs such as SoundStream (MOS ≈ 4.1 at 6 kbps) and EnCodec‑Lite, the 24 kHz version delivers higher fidelity at the same bitrate, thanks to its deeper encoder and the loss‑balancer. Traditional codecs (Opus, AAC) still win on ultra‑low‑bitrate scenarios (< 2 kbps) but suffer from noticeable artifacts above 4 kbps, where EnCodec excels.
Hardware Requirements
VRAM for inference
- Minimum: 2 GB GPU memory (sufficient for a single 1‑second chunk).
- Recommended: 4–6 GB for batch processing or streaming with larger look‑ahead windows.
GPU recommendations
- NVidia RTX 3080/3090, RTX A6000, or A100 – all provide the required Tensor cores for sub‑10 ms latency.
- AMD equivalents (e.g., Radeon RX 6900 XT) work via PyTorch’s ROCm backend, but real‑time performance may vary.
CPU & storage
- CPU: Modern 8‑core (Intel i7‑12700K, AMD Ryzen 7 5800X) – primarily for data loading and pre‑processing.
- Storage: Model files (~400 MB in safetensors format) plus a small cache for audio buffers; SSD recommended for low I/O latency.
During inference the model processes 1‑second audio chunks at ~30 fps on a single RTX 3090, making it suitable for live applications. Memory usage scales linearly with batch size; a batch of 8 chunks fits comfortably within 6 GB VRAM.
Use Cases
Primary applications
- Live voice‑chat compression for gaming and VR platforms.
- Real‑time streaming of podcasts or webinars over constrained networks.
- Pre‑processing step for speech‑recognition pipelines to reduce bandwidth without sacrificing accuracy.
- Music streaming services that want higher fidelity at low bitrates.
Real‑world examples
- Gaming: Integrate EnCodec into a Unity or Unreal Engine plugin to transmit player voice at 6 kbps with <10 ms latency.
- Telehealth: Use EnCodec to compress patient‑doctor audio calls, preserving diagnostic‑quality speech while reducing data costs.
- Content creation: Feed EnCodec’s latent codes into a generative model (e.g., MusicLM) for low‑bitrate music generation.
Because the model is exposed via the feature‑extraction pipeline tag, it can be chained with other Hugging Face models (e.g., Whisper, TTS) without custom preprocessing.
Training Details
EnCodec was trained for 300 epochs (≈ 600 k updates) on a mixture of speech, general audio, and music datasets. Each epoch consisted of 2 000 optimizer steps with a batch size of 64 one‑second audio clips. Training used the Adam optimizer (lr = 3 × 10⁻⁴, β₁ = 0.5, β₂ = 0.9) on 8 × NVIDIA A100 GPUs.
Datasets
- Speech: DNS Challenge 4, Common Voice.
- General audio: AudioSet (2 K subset), FSD50K.
- Music: Jamendo dataset.
The training pipeline employed a single multiscale spectrogram adversary to reduce artifacts and a loss‑balancer to stabilise convergence across heterogeneous data sources. The model can be fine‑tuned on domain‑specific audio (e.g., studio‑recorded music) by loading the pre‑trained weights and continuing training with a reduced learning rate (≈ 1 × 10⁻⁴) for a few thousand steps.
Licensing Information
The repository lists the license as unknown. In practice, this means the model is released under a non‑standard or proprietary license that has not been explicitly attached to the Hugging Face card. Users should:
- Review the original GitHub repository for any LICENSE file.
- Assume that commercial use is not guaranteed until permission is obtained from Meta AI.
- Provide attribution to “Meta AI – EnCodec (facebook/encodec_24khz)” in any derivative work or publication.
- Check the Hugging Face discussions for community‑reported licensing clarifications.
If a permissive license (e.g., MIT, Apache‑2.0) is later added, commercial deployment would be straightforward. Until then, treat the model as research‑only and seek legal counsel before integrating it into revenue‑generating products.