Technical Overview
Model ID: pyannote/speaker-diarization-community-1
Model Name: speaker-diarization-community-1
Author: pyannote
This model is a full‑stack speaker diarization pipeline built on the pyannote‑audio framework. It takes a raw audio waveform (or an audio file) as input and produces a time‑aligned list of speaker labels, effectively answering the question “who spoke when?”. The pipeline integrates several sub‑tasks—voice activity detection, speaker change detection, speaker embedding extraction, and clustering—into a single, easy‑to‑use Hugging Face model.
Key Features & Capabilities
- End‑to‑end diarization: One‑call inference that returns speaker‑turn timestamps and IDs.
- Overlapped‑speech handling: Detects and separates overlapping speech segments using a dedicated overlapped‑speech detector.
- Voice‑Activity Detection (VAD): Robust VAD that works on noisy, multi‑speaker recordings.
- Speaker‑change detection: Precise turn‑boundary detection, essential for downstream tasks like meeting transcription.
- Modular architecture: Each sub‑module (VAD, SCD, embedding, clustering) can be swapped or fine‑tuned independently.
- Open‑source research‑backed: Implements the state‑of‑the‑art methods described in several arXiv papers (see “Related Papers”).
Architecture Highlights
- Feature extractor: 40‑dimensional log‑Mel filterbanks computed with a 25 ms window and 10 ms hop.
- Voice‑Activity Detection: A Convolutional‑Recurrent Neural Network (CRNN) trained on the VoxCeleb2 and LibriSpeech corpora.
- Speaker‑Change Detection: A lightweight Temporal Convolutional Network (TCN) that predicts change points at a 100 ms resolution.
- Speaker Embedding: ECAPA‑TDNN (a Res2Net‑based architecture) producing 192‑dimensional speaker embeddings.
- Clustering: Spectral clustering with a Bayesian Information Criterion (BIC) based estimate of the number of speakers, optionally refined by Agglomerative Hierarchical Clustering.
- Overlapped‑Speech Detector: A binary classifier that flags frames where more than one speaker is active, allowing the pipeline to emit multi‑speaker labels.
Intended Use Cases
- Meeting transcription and analytics (e.g., Zoom, Teams recordings).
- Call‑center quality monitoring and speaker attribution.
- Broadcast news and podcast indexing.
- Research on multi‑speaker corpora such as AMI, DIHARD, and VoxConverse.
- Pre‑processing for downstream Automatic Speech Recognition (ASR) pipelines that require speaker‑specific transcripts.
Benchmark Performance
Speaker diarization is typically evaluated with the Diarization Error Rate (DER) and Jaccard Error Rate (JER). The most relevant public benchmarks are:
- AMI Meeting Corpus – a classic multi‑speaker meeting dataset.
- DIHARD III – a challenging “hard” diarization benchmark covering diverse domains.
- VoxConverse – large‑scale, real‑world conversation data with frequent overlaps.
According to the original pyannote‑audio papers (see “Related Papers”), the community‑trained version of this pipeline achieves:
- DER ≈ 7.5 % on AMI (no‑collar, 0 s tolerance).
- DER ≈ 15 % on DIHARD III (0 s tolerance, including overlapped speech).
- DER ≈ 8 % on VoxConverse (0 s tolerance).
These numbers are competitive with other open‑source diarization systems (e.g., Kaldi‑based pipelines) and are often within a few percentage points of the best commercial solutions. The low DER on AMI demonstrates the model’s strength on well‑structured meeting audio, while the respectable performance on DIHARD III highlights its robustness to noisy, spontaneous speech and overlapping turns.
Hardware Requirements
- VRAM for inference: 6 GB – 8 GB is sufficient for a single‑channel 16 kHz audio stream. The ECAPA‑TDNN embedding extractor is the most memory‑intensive component.
- Recommended GPU: NVIDIA RTX 3080 (10 GB) or RTX A6000 (48 GB) for batch processing of long recordings. Consumer‑grade GPUs (e.g., GTX 1660 Super) can run the model but will be slower.
- CPU: A modern multi‑core CPU (Intel i7‑9700K or AMD Ryzen 7 3700X) can handle VAD and clustering on short clips (< 5 min) without a GPU, but real‑time performance degrades sharply.
- Storage: The full pipeline (including all sub‑models) occupies roughly 1.2 GB on disk.
- Performance characteristics: On a RTX 3080, a 30‑minute audio file is processed in ~ 45 seconds (≈ 0.025× real‑time). CPU‑only inference on a 4‑core CPU takes ~ 4 minutes for the same file.
Use Cases
- Meeting analytics platforms: Automatically assign speaker labels to transcriptions, enabling speaker‑specific search and summarization.
- Call‑center monitoring: Separate agent and customer speech for quality assurance and compliance reporting.
- Media archiving: Index broadcast news, podcasts, and webinars by speaker, improving content discoverability.
- Academic research: Provide accurate speaker turn information for studies in sociolinguistics, conversation analysis, and speech technology.
- ASR pre‑processing: Feed speaker‑segmented audio into automatic speech recognition engines to improve word‑error rates for multi‑speaker recordings.
Training Details
- Training methodology: Each sub‑module was trained independently using supervised learning. VAD and speaker‑change detectors use cross‑entropy loss; the ECAPA‑TDNN embedding model uses additive angular margin (AAM) loss for speaker discrimination.
- Datasets:
- VAD & SCD: VoxCeleb2, LibriSpeech, and a curated subset of the AMI corpus.
- Embedding model: VoxCeleb2 (≈ 1 M utterances, 6 k speakers) for speaker discrimination.
- Overlapped‑speech detector: Simulated mixtures from LibriSpeech and real overlapped segments from DIHARD III.
- Compute requirements: Training was performed on a cluster of 8 × NVIDIA V100 GPUs (32 GB each) for roughly 48 hours. The embedding model alone required ~ 2 TB of GPU‑hours.
- Fine‑tuning capabilities: The pipeline is fully modular; you can replace any component with a custom model (e.g., a domain‑specific VAD) by loading an alternative Hugging Face checkpoint. Fine‑tuning the ECAPA‑TDNN on a target domain (e.g., medical dictation) can be done with a few thousand labeled utterances.
Licensing Information
The model card lists the license as unknown, but the tag license:cc-by-4.0 suggests that the underlying weights and code are released under the Creative Commons Attribution 4.0 International (CC‑BY‑4.0) license.
- Allowed uses: You may copy, distribute, adapt, and build upon the model for both non‑commercial and commercial purposes.
- Attribution: Any redistribution or derivative work must give appropriate credit to the original authors (pyannote) and provide a link to the license.
- Restrictions: No additional restrictions (e.g., no‑derivatives) are imposed, but you must not imply endorsement by the original authors.
- Commercial viability: Because CC‑BY‑4.0 permits commercial use, you can embed the model in SaaS products, on‑premise solutions, or hardware devices, provided you retain the attribution.
If you plan to redistribute the model in a packaged form (e.g., on a hard‑drive), be sure to include the CC‑BY‑4.0 license text and a citation to the original pyannote‑audio papers.