wav2vec2-base-960h

Technical Overview

What is this model? wav2vec2‑base‑960h is a self‑supervised speech representation model released by Meta (formerly Facebook). It is a pre‑trained acoustic encoder that has been fine‑tuned on 960 hours of the LibriSpeech corpus (16 kHz English speech) to perform end‑to‑end automatic speech recognition (ASR) via a CTC decoder. The model accepts raw audio waveforms and outputs token‑level logits that can be decoded into human‑readable transcriptions.

Key features and capabilities

• Raw‑audio input – no hand‑crafted feature extraction required.
• 16 kHz sampling rate (the model was trained on 16 kHz audio).
• CTC‑based decoding for fast, streaming‑friendly inference.
• High accuracy on LibriSpeech: 3.4 % WER on the “clean” test set and 8.6 % on the “other” test set.
• Compatible with 🤗 Transformers, PyTorch, TensorFlow and the safetensors format.
• Ready‑to‑use processor (Wav2Vec2Processor) that handles feature extraction, padding and token decoding.

Architecture highlights

• Base configuration: 12 transformer encoder layers, 768 hidden dimensions, 12 attention heads.
• Self‑supervised pre‑training using a contrastive loss on masked latent speech representations (wav2vec 2.0).
• Quantization layer that learns a discrete codebook for the latent space, enabling the contrastive objective.
• Fine‑tuning head: a linear projection to 32 k CTC tokens (including the blank token).

Intended use cases

• Real‑time transcription services (e.g., call‑center analytics, live captioning).
• Batch processing of large audio archives for searchable transcripts.
• Voice‑controlled applications that require low‑latency, on‑device inference.
• Research prototyping for low‑resource ASR or domain‑adaptation experiments.

Benchmark Performance

For ASR models, the most informative benchmark is Word Error Rate (WER) on the LibriSpeech test splits. The wav2vec2‑base‑960h model achieves:

These numbers are competitive with many supervised baselines while using a single model checkpoint. The “clean” WER of 3.4 % places the model among the top performers on the HF‑ASR leaderboard for English, and the “other” WER of 8.6 % demonstrates robustness to noisy or accented speech. Compared to earlier wav2vec 2.0 releases (e.g., the 300‑hour fine‑tuned variant), the 960‑hour version reduces WER by roughly 1–2 % absolute, confirming the benefit of more labeled data during fine‑tuning.

Hardware Requirements

VRAM for inference – The base model occupies ~300 MB of GPU memory when loaded in FP32. Using half‑precision (FP16) reduces the footprint to ~150 MB, allowing inference on consumer‑grade GPUs (e.g., RTX 3060 with 12 GB VRAM) with a comfortable margin for batch processing.

Recommended GPU specifications

• CUDA‑compatible GPU with at least 6 GB VRAM (RTX 2060, GTX 1080 Ti, or equivalent).
• For real‑time streaming, a GPU with ≥8 GB VRAM (RTX 3060 Ti, RTX 3070) is advisable to keep latency under 50 ms per second of audio.

CPU requirements – On CPU‑only inference, a modern 8‑core processor (e.g., Intel i7‑10700K or AMD Ryzen 7 3700X) can process ~1 second of audio per second using the torch.backends.cudnn.enabled = False fallback. Expect 2–3× slower throughput compared to GPU.

Storage needs – The model checkpoint (weights + tokenizer) is ~1 GB when stored as safetensors. Adding the optional configuration files and example audio samples brings the total download size to ~1.2 GB.

Performance characteristics – Inference speed scales linearly with batch size. A batch of 8 seconds of audio on an RTX 3080 processes in ~0.12 seconds (≈66 × real‑time). The model supports streaming inference by feeding overlapping windows of 20 ms with a stride of 10 ms.

Use Cases

Primary applications

• Live captioning for video conferencing platforms.
• Automatic transcription of podcasts, lectures, and meeting recordings.
• Voice‑controlled assistants that operate offline on edge devices.
• Speech‑to‑text preprocessing for downstream NLP tasks (e.g., sentiment analysis, intent detection).

Real‑world examples

• A call‑center analytics suite that ingests recorded calls, runs the model on a GPU‑accelerated backend, and stores searchable transcripts for compliance monitoring.
• An e‑learning platform that automatically generates subtitles for lecture videos, improving accessibility for hearing‑impaired learners.
• A mobile app that leverages on‑device inference (FP16) to provide instant voice notes without internet connectivity.

Training Details

Methodology – The model was first pre‑trained on 53 k hours of unlabeled audio using the wav2vec 2.0 contrastive loss. Afterwards, a supervised fine‑tuning stage was performed on the 960 hours of transcribed LibriSpeech data (clean + other splits). The fine‑tuning objective is Connectionist Temporal Classification (CTC) with a token vocabulary of 32 k characters.

Datasets

• Pre‑training: Unlabeled speech from the LibriVox corpus (≈53 k hours).
• Fine‑tuning: LibriSpeech “train‑clean‑100” + “train‑other‑500” (960 hours total).

Compute requirements – Pre‑training was carried out on 8× V100 GPUs for roughly 400 k steps. Fine‑tuning on 960 hours required a single V100 for ~50 k steps (≈12 hours). The model can be fine‑tuned further on domain‑specific data using the same CTC head.

Fine‑tuning capabilities – Users can adapt the model to new languages, accents, or vocabularies by:

• Replacing the CTC token set (e.g., adding domain‑specific symbols).
• Continuing training on a small labeled corpus (as little as 10 minutes of speech) while keeping the encoder frozen.
• Employing data‑augmentation techniques such as speed perturbation or SpecAugment to improve robustness.

Related Papers

The core research behind this model is described in the paper:

• wav2vec 2.0: Learning the Structure of Speech from Raw Audio – Baevski, Zhou, Mohamed, Auli (2020).

This work introduced the contrastive self‑supervised objective, latent‑space masking, and quantization that enable high‑quality speech representations without any hand‑crafted features. Subsequent studies on low‑resource ASR and multilingual extensions build directly on the wav2vec 2.0 architecture.

Licensing Information

The model card lists the apache‑2.0 license for the weights and code, while the overall repository tag shows license: unknown. In practice, the Apache 2.0 license applies to the released checkpoint and associated processor files.

Commercial use – Apache 2.0 is a permissive open‑source license that explicitly allows commercial exploitation, redistribution, and modification, provided that:

• A copy of the license is included with any distribution.
• Any modifications are clearly marked.
• Patents granted by contributors are licensed for use.

No royalty fees or additional permissions are required for commercial products that embed the model. However, users should verify that any downstream data (e.g., LibriSpeech) complies with its own licensing terms.