hubert-large-ll60k

Technical Overview

What is this model? hubert-large-ll60k is Facebook’s large‑scale, self‑supervised speech representation model. Trained on 60 000 hours of 16 kHz audio from the Libri‑Light corpus, it learns rich acoustic embeddings without any textual supervision. The model is exposed through the feature‑extraction pipeline, meaning it converts raw waveform input into high‑dimensional feature vectors that can be downstream for automatic speech recognition (ASR), speaker verification, speech synthesis, or any task that benefits from a robust speech encoder.

Key features & capabilities

• Large‑scale encoder with ~300 M parameters (the “large” variant of HuBERT).
• Works on 16 kHz mono audio; no resampling needed if your data matches this rate.
• Self‑supervised training via masked prediction on clustered pseudo‑labels, providing both acoustic and implicit language modeling.
• Compatible with Hugging Face HubertForCTC for end‑to‑end speech‑to‑text fine‑tuning.
• Supports PyTorch, TensorFlow and ONNX export for flexible deployment.

Architecture highlights

• Based on the Transformer encoder architecture introduced in wav2vec 2.0, but with a deeper stack (24 layers) and larger hidden size (1024).
• Input front‑end: a 7‑layer convolutional feature extractor that converts raw waveforms into 512‑dimensional acoustic frames.
• Masked prediction loss: random spans of the latent sequence are masked and the model learns to predict the cluster IDs generated by an offline k‑means teacher.
• Two‑stage clustering: an initial 100‑cluster k‑means step followed by a refined 500‑cluster step, which dramatically improves label consistency.

Intended use cases

• Pre‑training backbone for ASR systems (fine‑tune with CTC or sequence‑to‑sequence heads).
• Feature extractor for speaker diarization, emotion detection, and language identification.
• Audio compression or generative tasks where a compact, high‑quality representation is required.

Benchmark Performance

HuBERT‑large has been evaluated on the standard Librispeech and Libri‑Light benchmarks. The README cites the original paper (arXiv:2106.07447) which reports that the model matches or surpasses wav2vec 2.0 on fine‑tuning subsets ranging from 10 minutes to the full 960 hours. In particular, a 1 B‑parameter HuBERT variant achieved up to a 19 % relative WER reduction on the “dev‑other” set and 13 % on “test‑other”. While the < < large‑ll60k model is smaller, it still delivers state‑of‑the‑art performance in low‑resource fine‑tuning scenarios (e.g., 1 hour of labeled data) and is competitive with wav2vec 2.0‑large on the 100‑hour subset.

Why these benchmarks matter Librispeech and Libri‑Light are the de‑facto standards for evaluating self‑supervised speech encoders because they provide a wide range of labeled and unlabeled data, allowing researchers to measure both representation quality and downstream ASR performance. Improvements on these datasets translate directly to better real‑world transcription accuracy, especially in noisy or low‑resource environments.

Hardware Requirements

VRAM for inference The large encoder (≈300 M parameters) comfortably fits on a single 12 GB GPU (e.g., NVIDIA RTX 3060) when using batch size = 1. For batch processing or real‑time streaming, a 16 GB GPU (RTX 3070/3080) is recommended to keep latency below 30 ms per second of audio.

Recommended GPU specifications

• CUDA ≥ 11.3, cuDNN ≥ 8.2
• GPU memory: 12 GB minimum; 16 GB+ for high‑throughput workloads
• GPU compute: 8 TFLOPs (FP16) or higher for optimal masked‑prediction speed

CPU & storage

• CPU: 8‑core modern processor (e.g., Intel i7‑12700K or AMD Ryzen 7 5800X) for preprocessing and data loading.
• RAM: 16 GB minimum; 32 GB recommended when fine‑tuning on large datasets.
• Disk: The model checkpoint is ~1.2 GB; store on SSD for fast loading.

Performance characteristics On a RTX 3080 (10 GB) the model processes ~2 seconds of audio per GPU second (≈0.5× real‑time) in FP16 mode. Using TorchScript or ONNX can push this to >3× real‑time on the same hardware.

Use Cases

Primary applications

• Automatic Speech Recognition (ASR) – Fine‑tune HubertForCTC on domain‑specific transcripts (e.g., call‑center audio) for high‑accuracy speech‑to‑text.
• Speaker Verification & Diarization – Use the extracted embeddings as inputs to a cosine‑similarity or clustering pipeline.
• Emotion & Intent Detection – Feed the features into a lightweight classifier for affective computing.
• Audio Compression & Generation – Leverage the latent representation for downstream generative models such as neural codecs.

Real‑world examples

• Transcribing medical dictations where a 1‑hour fine‑tuning set already yields >90 % word‑accuracy.
• Building a multilingual voice assistant that re‑uses the same encoder across languages after a short fine‑tune.
• Analyzing call‑center recordings for compliance monitoring without storing raw audio (store only embeddings).

Training Details

Methodology HuBERT‑large follows a three‑step training regime:

1. Feature extraction – 7‑layer convolutional encoder converts raw 16 kHz waveform into 512‑dimensional acoustic frames.
2. Clustering – An offline k‑means algorithm first creates 100 pseudo‑labels on the raw features. After an initial pre‑training pass, a second k‑means with 500 clusters refines the labels.
3. Masked prediction – Random spans (≈15 % of frames) are masked; the Transformer encoder learns to predict the cluster ID of each masked frame using a cross‑entropy loss.

Datasets The model was trained on the Libri‑Light corpus (≈60 000 hours of English speech). No textual transcripts were used during pre‑training.

Compute Training was performed on a cluster of 8 × NVIDIA V100 GPUs (32 GB each) for roughly 400 k steps, consuming ~2 M GPU‑hours. The large variant required mixed‑precision (FP16) to fit the 300 M‑parameter model in GPU memory.

Fine‑tuning capabilities The checkpoint can be loaded into HubertForCTC (or HubertForSequenceClassification) and fine‑tuned on as little as 10 minutes of labeled audio, achieving competitive WER on Librispeech. Users should supply a tokenizer (e.g., a character‑level tokenizer) because the original model has no built‑in tokenization.

Related Papers

The core research behind this model is presented in the paper “Self‑Supervised Representation Learning for Speech Recognition, Generation, and Compression” (arXiv:2106.07447). The authors—Wei‑Ning Hsu, Benjamin Bolte, Yao‑Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed—introduce the Hidden‑Unit BERT (HuBERT) framework, which combines offline clustering with masked prediction. This work builds on prior self‑supervised audio models such as wav2vec 2.0 and introduces a two‑stage clustering pipeline that dramatically improves label consistency and downstream WER.

Licensing Information

The model card lists the license as unknown, but the README specifies an Apache‑2.0 license for the underlying research code and pretrained weights. Apache‑2.0 is a permissive open‑source license that allows commercial use, modification, and distribution, provided that:

• A copy of the license is included with any redistribution.
• Significant changes are documented.
• Trademark usage (e.g., “Facebook”) is avoided without permission.

If you plan to embed the model in a proprietary product, you can do so safely under Apache‑2.0. However, double‑check the exact license file in the repository because some third‑party components (e.g., Fairseq) may have additional requirements.