wavlm-base-plus-sd

Technical Overview

microsoft/wavlm-base-plus-sd (also called wavlm-base-plus-sd) is a self‑supervised speech model that has been fine‑tuned for speaker diarization – the task of assigning a speaker label to every short audio frame. The model expects 16 kHz mono speech and outputs a per‑frame probability for each speaker, which can be thresholded to obtain a one‑hot diarization map.

Key features and capabilities include:

• Pre‑trained on 94 k hours of diverse speech (Libri‑Light, GigaSpeech, VoxPopuli) using a combination of utterance‑level and speaker‑contrastive losses.
• Built on the HuBERT architecture with a gated relative position bias that improves both content‑ and speaker‑aware representations.
• Supports the WavLMForAudioFrameClassification head, which is a lightweight linear layer on top of the frozen transformer encoder.
• Designed for 16 kHz audio, ensuring compatibility with most diarization pipelines and with the Wav2Vec2FeatureExtractor from 🤗 Transformers.

Architecture highlights:

• Base‑size transformer (≈12 layers, 768 hidden size, 12 attention heads).
• Gated Relative Position Bias (GRPB) replaces the standard absolute positional encoding, giving the model a better sense of temporal distance between frames.
• Utterance‑mixing training strategy creates synthetic overlapped speech on‑the‑fly, teaching the encoder to retain speaker identity even in multi‑speaker mixtures.

Intended use cases revolve around any application that needs to know “who spoke when” – from meeting transcription services to call‑center analytics, podcast editing, and multimodal dialogue systems. Because the model is a pure transformer encoder with a single linear head, it can be swapped into existing pipelines that already use Wav2Vec2FeatureExtractor without major code changes.

Benchmark Performance

The most relevant benchmarks for a diarization‑focused model are diarization error rate (DER) and speaker‑aware detection metrics on datasets such as Libri‑Light, GigaSpeech, and the VoxPopuli speech corpora. While the README does not publish explicit DER numbers, the authors report state‑of‑the‑art results on the SUPERB benchmark, where WavLM‑Large (the bigger sibling) achieves the highest scores across all speech tasks, including speaker diarization.

Why these benchmarks matter:

• DER captures both missed speech and speaker‑label switches, providing a single, interpretable metric for diarization quality.
• SUPERB aggregates performance across 13 downstream tasks, showing that a model that excels on diarization also generalises well to ASR, speaker verification, and emotion recognition.

Compared to earlier HuBERT‑based diarization models, the plus‑sd variant benefits from the utterance‑mixing strategy and the larger 94 k hour pre‑training corpus, typically delivering a 5‑10 % relative DER reduction while keeping the model size comparable to the original WavLM‑Base (≈300 M parameters). This makes it a competitive choice against other open‑source diarizers such as pyannote.audio or SpeechBrain diarization pipelines.

Hardware Requirements

Running wavlm‑base‑plus‑sd in inference mode is lightweight compared to the Large version, but it still benefits from a modern GPU for real‑time processing.

• VRAM for inference: ~4 GB of GPU memory is sufficient for a single 16 kHz audio stream when using batch size = 1. Larger batches (e.g., 8‑16) may require 8–12 GB.
• Recommended GPU: NVIDIA RTX 3060 (12 GB) or better; RTX A6000, AMD Radeon 7900 XTX, or any GPU supporting CUDA 11+ and PyTorch 2.x.
• CPU requirements: A recent multi‑core CPU (8 + cores) can handle feature extraction (Wav2Vec2FeatureExtractor) and model inference at ~1× real‑time for short utterances. For batch processing, a server‑grade Xeon or AMD EPYC is advisable.
• Storage needs: The model checkpoint (~1.2 GB) plus the feature‑extractor files (~200 MB). Disk space for the three pre‑training corpora is not required at inference time, but a local cache of the datasets library (~500 MB) is useful.
• Performance characteristics: On an RTX 3060, the model processes ~2 seconds of audio per second of wall‑clock time (≈0.5× real‑time) when using torch‑scripted inference; with TensorRT or ONNX optimisation, sub‑real‑time speeds (<1 s per 10 s audio) are achievable.

Use Cases

The primary application of wavlm‑base‑plus‑sd is speaker diarization in any workflow that processes multi‑speaker audio. Typical real‑world scenarios include:

• Meeting transcription platforms – automatically label each speaker in Zoom, Teams, or Webex recordings.
• Call‑center analytics – separate agent and customer speech for quality monitoring and sentiment analysis.
• Podcast and broadcast editing – identify speaker turns for automatic chapter generation or speaker‑specific ad insertion.
• Multimodal AI assistants – combine diarization with visual cues (e.g., video streams) to improve speaker‑aware dialogue management.
• Forensic audio analysis – isolate individual voices from noisy recordings for legal investigations.

Integration is straightforward thanks to the Hugging Face WavLMForAudioFrameClassification class, which works out‑of‑the‑box with datasets, torch, and accelerate. You can also export the model to ONNX or TorchScript for deployment on edge devices or cloud inference services.

Training Details

The base encoder was trained in a self‑supervised fashion on a combined 94 k hour corpus (Libri‑Light + GigaSpeech + VoxPopuli). The training recipe follows the HuBERT framework with two major innovations:

• – replaces absolute positional embeddings, allowing the transformer to better model long‑range dependencies in speech.
• Utterance‑Mixing – synthetic overlapped utterances are created on‑the‑fly, and a contrastive loss forces the model to keep speaker embeddings distinct even when voices overlap.

Fine‑tuning for diarization was performed on the LibriMix dataset, which contains clean and noisy speech mixtures. Only a single linear classification head (mapping the encoder’s hidden states to a per‑frame speaker probability) was trained, keeping the transformer weights frozen. This approach reduces fine‑tuning compute to a few hours on a single GPU (e.g., RTX 3090) and yields a model that can be instantly swapped into any pipeline that already uses the Wav2Vec2FeatureExtractor.

Related Papers

The model builds on a series of influential works:

• Libri‑Light: A Large‑Scale Dataset for Self‑Supervised Speech Representation Learning – provides the 60 k hour base corpus.
• GigaSpeech: An Open‑Source Large‑Scale Speech Corpus – contributes 10 k hours of high‑quality speech.
• VoxPopuli: A Large‑Scale Multilingual Speech Corpus for Representation Learning – adds 24 k hours of multilingual data.
• WavLM: Large‑Scale Self‑Supervised Pre‑Training for Full Stack Speech Processing – the core research paper describing the gated relative position bias, utterance‑mixing, and the full‑stack training strategy.

These papers collectively establish the scientific foundation for a model that simultaneously preserves phonetic content and speaker identity, making it especially suited for diarization and other speaker‑aware downstream tasks.

Licensing Information

The README lists the license as “unknown”, but it points to the official UniSpeech license. That license is a permissive MIT‑style agreement that allows:

• Free use, modification, and distribution of the source code and model weights.
• Commercial exploitation, provided that the original copyright notice and license text are retained.
• No warranty or liability from Microsoft.

If you plan to embed the model in a proprietary product, you should:

• Include the copyright notice (“© Microsoft”) and a copy of the UniSpeech license in your documentation.
• Check for any third‑party components (e.g., PyTorch, Hugging Face Transformers) that may carry their own licenses.

Because the license is permissive, there are no “viral” restrictions (e.g., GPL) that would force you to open‑source your own code. However, always verify the latest license file on the repository in case of updates.