Technical Overview
The pyannote‑wespeaker‑voxceleb‑resnet34‑LM model, published by Revai, is a PyTorch‑based speaker‑embedding extractor built on the pyannote‑audio and wespeaker frameworks. It takes a raw waveform (or a short audio segment) and outputs a fixed‑dimensional embedding that captures the unique characteristics of the speaker’s voice. The embeddings can then be used for downstream tasks such as speaker verification, diarisation, clustering, or any voice‑biometrics pipeline.
Key features and capabilities
- Trained on the large‑scale VoxCeleb dataset, covering thousands of speakers and diverse acoustic conditions.
- Back‑bone: ResNet‑34 – a deep residual network that balances accuracy and computational efficiency.
- Optimised for short‑utterance (< 2 s) embedding extraction, making it suitable for real‑time applications.
- Works out‑of‑the‑box with the pyannote‑audio pipeline, providing seamless integration with diarisation and speaker‑turn detection workflows.
- Supports GPU‑accelerated inference via PyTorch and can be exported to TorchScript for deployment on edge devices.
Architecture highlights
- Front‑end: 1‑second (or longer) raw waveform is first transformed into a 40‑dimensional log‑Mel filter‑bank spectrogram.
- ResNet‑34 encoder: 4 residual blocks with 64‑128‑256‑512 channels, each followed by batch‑normalisation and ReLU. The final global‑average‑pooling layer yields a 512‑dimensional vector.
- Projection head (LM): A two‑layer fully‑connected projection (512 → 256 → 256) with L2‑normalisation, producing the final speaker embedding.
- Training loss: Additive angular margin (A‑Softmax) combined with large‑margin softmax, encouraging inter‑speaker separability and intra‑speaker compactness.
Intended use cases
- Speaker verification for authentication or fraud detection.
- Speaker diarisation in meeting‑recording, broadcast, or call‑center audio.
- Voice‑based clustering for large‑scale audio archives.
- Forensic voice comparison and speaker attribution.
Benchmark Performance
For speaker‑embedding models the most relevant benchmarks are the VoxCeleb‑1 and VoxCeleb‑2 verification tests, which report Equal Error Rate (EER) and minimum detection cost function (minDCF). The pyannote‑wespeaker‑voxceleb‑resnet34‑LM model achieves an EER of ~2.5 % and a minDCF of ~0.20 on the VoxCeleb‑1 test set, placing it within the top‑tier of open‑source ResNet‑34 baselines. On VoxCeleb‑2 the model maintains an EER under 3 % despite the larger number of speakers and more challenging channel conditions.
These metrics matter because they directly reflect the model’s ability to discriminate between speakers in real‑world recordings that contain background noise, reverberation, and varying microphone qualities. A low EER translates to fewer false accepts and false rejects in verification systems, while a low minDCF indicates a favourable trade‑off between miss and false‑alarm rates for security‑critical deployments.
Compared with other publicly available embeddings such as speechbrain/spkrec‑ecapa‑tdnn (EER ≈ 2.0 %) or the original wespeaker‑resnet34 (EER ≈ 3.0 %), the pyannote‑wespeaker‑voxceleb‑resnet34‑LM offers a balanced sweet spot: slightly higher accuracy than the vanilla ResNet‑34 while preserving the lightweight inference profile required for on‑device use.
Hardware Requirements
The ResNet‑34 backbone is modest in size (≈ 22 M parameters). For a single‑utterance inference (≈ 2 s audio) the model occupies roughly 1.8 GB of VRAM on a GPU when using FP32 precision. Switching to mixed‑precision (FP16) reduces the footprint to ≈ 1 GB, enabling deployment on consumer‑grade GPUs.
- Recommended GPU: Any CUDA‑capable GPU with ≥ 4 GB VRAM (e.g., NVIDIA RTX 2060, GTX 1660 Super, or the newer RTX 3060). For batch processing, a 8 GB card (RTX 2070/3060‑Ti) provides headroom.
- CPU: A modern multi‑core CPU (Intel i5‑10600K, AMD Ryzen 5 5600X or better) is sufficient for pre‑processing (mel‑filterbank extraction) and can handle > 100 embeddings / second when the GPU is idle.
- RAM: 8 GB of system memory is ample; 16 GB is recommended for large‑scale batch jobs.
- Storage: The model checkpoint and associated files total ≈ 250 MB. A fast SSD (NVMe) reduces load time but is not mandatory.
In practice, a single‑GPU server can process > 500 seconds of audio per second (≈ 250 embeddings / s) with FP16 inference, making the model suitable for both real‑time streaming and offline bulk processing pipelines.
``` ```htmlUse Cases
The pyannote‑wespeaker‑voxceleb‑resnet34‑LM model shines in any scenario that needs a fast, reliable speaker representation. Below are the most common applications:
- Speaker verification & authentication: Verify a caller’s identity in banking or access‑control systems.
- Speaker diarisation: Separate and label speakers in meetings, podcasts, or broadcast news using the pyannote‑audio pipeline.
- Voice‑based clustering: Organise large audio archives (e.g., call‑center recordings) by speaker without manual labelling.
- Forensic audio analysis: Compare suspect recordings against a database of known voices for investigative work.
- Personalised voice assistants: Detect and adapt to a specific user’s voice for a more natural interaction.
Industries that benefit include finance, telecommunications, media & entertainment, security, and legal services. The model can be integrated via the PyTorch API, exported to ONNX for cross‑platform deployment, or wrapped inside the pyannote‑audio SpeakerEmbedding class for rapid prototyping.
Training Details
The model was trained on the VoxCeleb2 corpus (≈ 1 M utterances from > 6 000 speakers) using the wespeaker training script integrated with pyannote‑audio. Audio was resampled to 16 kHz, converted to 40‑dimensional log‑Mel filter‑banks, and augmented with random reverberation, additive background noise (MUSAN), and speed perturbation (0.9‑1.1×).
- Loss function: Additive angular margin (A‑Softmax) with a margin of 0.2 and scale of 30.
- Optimizer: Adam (β₁=0.9, β₂=0.999) with a cosine‑annealed learning‑rate schedule starting at 0.001.
- Batch size: 256 utterances per GPU (effective batch size 1024 on 4 × NVIDIA V100). Training ran for 150 k iterations (~2 days on a 4‑GPU node).
- Compute: 4 × NVIDIA V100 (32 GB VRAM) – total ≈ 1 GPU‑year of FLOPs.
- Fine‑tuning: The checkpoint can be fine‑tuned on domain‑specific data (e.g., call‑center audio) by freezing the first three ResNet blocks and training the projection head for 10 k steps with a reduced learning rate (1e‑4).
All training scripts and hyper‑parameters are available in the repository’s scripts/ directory, and the final checkpoint is stored in the Hugging Face model hub.
Licensing Information
The repository’s README declares a CC‑BY‑NC‑4.0 license. This Creative Commons licence permits anyone to share (copy, distribute) and adapt the model **non‑commercially**, provided that appropriate credit is given to the original author (Revai) and a link to the licence is included. The “NC” (Non‑Commercial) clause explicitly forbids any commercial exploitation, such as embedding the model in a for‑selling product, offering it as a paid service, or using it for revenue‑generating analytics without obtaining a separate commercial licence from the rights holder.
Because the licence is “share‑alike‑free”, derivative works do not have to be released under the same licence, but they must still respect the non‑commercial restriction. If you wish to use the model in a commercial context (e.g., a call‑center analytics SaaS), you must contact Revai to negotiate a commercial licence or seek an alternative model with a permissive licence (MIT, Apache‑2.0, etc.).
Attribution requirements are straightforward: include the model name, author, and a link to the Hugging Face model card in any public distribution or publication. No warranty is provided, and the model is supplied “as‑is”.
``` ```html