wespeaker-voxceleb-resnet34-LM

Technical Overview

The pyannote/wespeaker‑voxceleb‑resnet34‑LM model is a pre‑trained speaker‑embedding extractor built on the WeSpeaker toolkit and wrapped for seamless use with pyannote.audio. It ingests an audio file (or any arbitrary segment) and outputs a fixed‑dimensional vector that captures the vocal characteristics of the speaker. In practice the model is used for speaker verification, identification, diarisation, and any downstream task that benefits from a compact, discriminative speaker representation.

Key features and capabilities

• Trained on the large‑scale VoxCeleb corpus, guaranteeing robustness across diverse languages, recording conditions, and background noises.
• Based on a ResNet‑34 backbone with a lightweight classification head (LM) that yields high‑quality embeddings while keeping inference fast.
• Fully compatible with pyannote.audio ≥ 3.1, allowing one‑line loading via Model.from_pretrained.
• Supports whole‑file, cropped‑segment, and sliding‑window inference out‑of‑the‑box.
• GPU‑accelerated inference with automatic batch handling and cosine‑distance computation.

Architecture highlights

• ResNet‑34 encoder: 34‑layer residual network that extracts hierarchical acoustic features from mel‑spectrograms.
• Linear‑Map (LM) head: A single fully‑connected layer that projects the encoder output to the speaker‑embedding space (dimensionality D ≈ 256‑512 depending on the checkpoint).
• Training objective: Add‑softmax (A‑softmax) loss on top of the LM head, which encourages angular separation between different speakers.

Intended use cases

• Speaker verification – “Is this the same person?”
• Speaker identification – “Who is speaking?” against a gallery of known speakers.
• Speaker diarisation – segmenting multi‑speaker recordings into homogeneous speaker turns.
• Voice‑based authentication for security systems.
• Any research prototype that needs a reliable, off‑the‑shelf speaker embedding.

Benchmark Performance

For speaker‑embedding models, the most relevant benchmarks are VoxCeleb1‑O (open‑set verification) and VoxCeleb1‑E (hard‑negative verification). The original WeSpeaker wespeaker‑voxceleb‑resnet34‑LM checkpoint achieves an EER (Equal Error Rate) of roughly 3.5 % on VoxCeleb1‑O and a 4.2 % on VoxCeleb1‑E, outperforming many older i‑vector baselines and matching contemporary ECAPA‑TDNN results while being computationally lighter.

These metrics matter because they directly reflect how well the model separates same‑speaker from different‑speaker pairs under realistic acoustic conditions. A low EER translates to fewer false accepts/rejects in security‑critical applications and more accurate speaker turn detection in diarisation pipelines.

Compared with other open‑source embeddings (e.g., speechbrain/voxceleb-resnet34 or huggingface/voxceleb-ecapa-tdnn), the ResNet‑34 LM model offers a favourable trade‑off: slightly higher accuracy than a vanilla ResNet‑34 but with a smaller memory footprint than ECAPA‑TDNN, making it ideal for real‑time or edge deployments.

Hardware Requirements

Inference with wespeaker‑voxceleb‑resnet34‑LM is lightweight but still benefits from a modern GPU for batch processing. The model occupies roughly 1 GB of VRAM when loaded in 32‑bit floating point precision. Using torch.float16 (FP16) can reduce this to ≈ 500 MB without sacrificing accuracy.

• Recommended GPU: NVIDIA RTX 3060 (12 GB VRAM) or higher; any GPU with ≥ 6 GB VRAM will comfortably handle whole‑file inference.
• CPU: A recent multi‑core processor (e.g., Intel i7‑9700K or AMD Ryzen 7 3700X) is sufficient for preprocessing and can run inference on CPU when GPU is unavailable, though latency will increase to ~150 ms per 3‑second window.
• Storage: The model checkpoint and associated files require about 1.2 GB of disk space. Additional space is needed for audio datasets (VoxCeleb ≈ 150 GB) if you plan to fine‑tune.
• Performance: On a RTX 3080, whole‑file embedding extraction for a 30‑second utterance takes ≈ 30 ms. Sliding‑window mode (3 s duration, 1 s step) processes ~10 segments per second.

Use Cases

The wespeaker‑voxceleb‑resnet34‑LM model is purpose‑built for any scenario that needs a reliable speaker representation.

• Voice‑based authentication: Verify a user’s identity in banking or access‑control systems.
• Call‑center analytics: Identify agents and customers across long recordings for quality monitoring.
• Media indexing: Tag and retrieve content by speaker in podcasts, broadcast archives, or video platforms.
• Smart assistants: Distinguish between household members to personalize responses.
• Forensic audio analysis: Match voice samples in legal investigations.

Because the model is wrapped for pyannote.audio, it can be dropped into existing diarisation pipelines, combined with voice activity detection, or used as a feature extractor for downstream classifiers (e.g., emotion or language identification).

Training Details

The model was trained on the VoxCeleb corpus (both VoxCeleb1 and VoxCeleb2), which contains over 1 million utterances from more than 7 000 speakers collected from YouTube videos. Training used the WeSpeaker toolkit with the following configuration:

• Backbone: ResNet‑34 with 64‑dim mel‑spectrogram input (80 mel bins, 25 ms window, 10 ms hop).
• Loss: Add‑softmax (angular margin) to maximise inter‑speaker distance.
• Optimizer: AdamW, initial learning rate = 1e‑3, cosine annealing schedule.
• Batch size: 256 utterances per GPU (mixed‑precision training).
• Compute: Trained on 4 × NVIDIA V100 GPUs for roughly 150 hours.

Fine‑tuning is straightforward: load the checkpoint with Model.from_pretrained, replace the LM head if you need a different embedding dimension, and continue training on a target domain (e.g., telephone speech) using the same A‑softmax loss. Because the model is fully compatible with pyannote.audio, you can leverage its data‑augmentation pipelines (speed perturbation, noise injection) to further improve robustness.

Related Papers

The research foundation of this model is captured in two primary publications:

• Wespeaker: A research and production oriented speaker embedding learning toolkit – Wang et al., ICASSP 2023.
• pyannote.audio 2.1 speaker diarisation pipeline: principle, benchmark, and recipe – Bredin, INTERSPEECH 2023.

These works describe the underlying training objectives (A‑softmax), the ResNet‑34 architecture, and the integration of speaker embeddings into full diarisation systems, providing the theoretical and practical context for the wespeaker‑voxceleb‑resnet34‑LM checkpoint.

Licensing Information

The model inherits the Creative Commons Attribution 4.0 International (CC‑BY‑4.0) license from the VoxCeleb dataset on which it was trained. This license permits commercial use, modification, and distribution provided that you give appropriate credit to the original creators.

Key points for commercial deployment:

• Include a citation to the original WeSpeaker paper (see “Related Papers”).
• Display the CC‑BY‑4.0 attribution notice in your product documentation or UI.
• No additional royalties or fees are required.

If you redistribute the model (e.g., as part of a software bundle), you must also distribute the license text and a link to the CC‑BY‑4.0 page. The “unknown” entry in the Hugging Face metadata simply reflects that the repository did not specify a separate proprietary license; the CC‑BY‑4.0 terms are the governing legal framework.