emotion-recognition-wav2vec2-IEMOCAP

Technical Overview

The speechbrain/emotion-recognition-wav2vec2-IEMOCAP model is a fine‑tuned wav2vec2‑base encoder that transforms raw audio into a probability distribution over six basic emotions (e.g., anger, happiness, sadness, neutral, etc.). Built on the SpeechBrain toolkit, it operates on 16 kHz mono recordings and automatically handles resampling and channel selection during inference.

Key capabilities include:

• End‑to‑end audio classification – no hand‑crafted features are required; the model learns directly from the waveform.
• Attentive statistical pooling – a learnable pooling layer aggregates frame‑level wav2vec2 embeddings into a fixed‑size utterance representation.
• Additive Margin Softmax (AM‑Softmax) loss – encourages larger inter‑class margins, improving discriminability of emotion classes.
• Cosine‑distance speaker verification – the same architecture can be repurposed for speaker embedding extraction, demonstrating its versatility.

Architecturally, the system combines:

• A pre‑trained wav2vec2‑base encoder (12 transformer blocks, 768‑dimensional hidden size) that supplies rich acoustic embeddings.
• A stack of convolutional and residual blocks that refine the wav2vec2 output.
• An attentive statistical pooling layer that learns to weight frames based on their relevance to the target emotion.
• A final linear classifier trained with AM‑Softmax.

Intended use cases span any scenario where automatic detection of affective state from speech is valuable: call‑center analytics, mental‑health monitoring, interactive voice assistants, and multimodal emotion analysis pipelines. Because the model is packaged as a Hugging Face audio‑classification pipeline, it can be dropped into Python scripts, web services, or edge‑device runtimes with minimal boilerplate.

Benchmark Performance

For emotion‑recognition systems, the primary benchmark is accuracy on a held‑out test split of the target dataset. The IEMOCAP corpus, a widely‑used benchmark for affective speech, provides a realistic mix of scripted and improvised dialogues with a range of speakers, emotions, and acoustic conditions.

According to the model card, the 19‑10‑21 release achieved an average accuracy of 78.7 % (average across folds: 75.3 %). This places the model among the top‑performing wav2vec2‑based classifiers for IEMOCAP, outperforming classic MFCC‑+ SVM pipelines (typically 60‑70 % accuracy) and approaching the performance of larger transformer‑based systems that require substantially more compute.

Why these numbers matter:

• High accuracy translates directly into more reliable downstream analytics (e.g., sentiment dashboards).
• Consistent performance across speakers indicates robustness to speaker variability, a common challenge in real‑world deployments.
• Benchmarks on IEMOCAP are a de‑facto standard, making it easy to compare against other published models.

Hardware Requirements

The model’s backbone (wav2vec2‑base) contains roughly 95 M parameters. For inference, the memory footprint is modest compared with larger wav2vec2‑large variants.

• VRAM for inference – ~2 GB of GPU memory is sufficient for a single‑utterance batch (batch size = 1). Larger batches can be processed on 4 GB+ GPUs.
• Recommended GPU – NVIDIA RTX 2060 / GTX 1660 Super or newer (CUDA ≥ 11.0). For batch processing, a RTX 3060 (12 GB) or A100 (40 GB) provides ample headroom.
• CPU requirements – Any modern x86‑64 CPU can run the model in real time for short utterances (< 5 s). A 4‑core CPU (e.g., Intel i5‑10400) with AVX2 support yields ~30 ms latency per utterance.
• Storage – The model checkpoint (~400 MB) plus the wav2vec2 base weights (~350 MB) total under 1 GB. Including the repository and optional training logs, allocate at least 2 GB of disk space.
• Performance characteristics – On a RTX 2070, end‑to‑end latency (audio loading → probability output) is ≈ 15 ms for a 3‑second clip. CPU‑only inference on a 2.5 GHz quad‑core is ≈ 120 ms.

Use Cases

The model is engineered for any application that needs to infer a speaker’s emotional state directly from audio. Typical scenarios include:

• Customer‑service analytics – Automatic tagging of call‑center recordings to flag angry or frustrated callers for supervisory review.
• Digital health & mental‑wellness apps – Continuous monitoring of voice‑based diaries to detect mood shifts, enabling early intervention.
• Interactive voice assistants – Adjusting response tone or offering empathy when the user sounds upset.
• Multimodal emotion research – Providing a reliable audio‑only baseline that can be fused with video or physiological signals.
• Education technology – Measuring student engagement and frustration during spoken‑language exercises.

Integration is straightforward via the foreign_class interface described in the README, or by using the Hugging Face pipeline("audio-classification") API. The model can be wrapped in a REST endpoint, deployed on a serverless function, or embedded on an edge device that runs PyTorch.

Training Details

Training was performed with the SpeechBrain recipe recipes/IEMOCAP/emotion_recognition. The pipeline follows these steps:

1. Data preparation – Audio files from IEMOCAP are resampled to 16 kHz, normalized, and split into training/validation/test folds.
2. Feature extraction – The wav2vec2‑base encoder (pre‑trained on LibriSpeech) processes the raw waveform, producing a sequence of 768‑dimensional embeddings.
3. Model head – Convolutional + residual layers refine these embeddings, followed by attentive statistical pooling that yields a fixed‑size utterance vector.
4. Loss function – Additive Margin Softmax (AM‑Softmax) with a margin of 0.35 encourages larger separation between emotion classes.
5. Optimization – AdamW optimizer, learning‑rate warm‑up for 5 k steps, then cosine decay. Training runs for 100 k updates (≈ 30 epochs).

The training was executed on a single NVIDIA V100 GPU (16 GB VRAM) and took roughly 12 hours. The resulting checkpoint (≈ 400 MB) is uploaded to Hugging Face and can be fine‑tuned further on domain‑specific data by simply swapping the train_with_wav2vec2.py script and adjusting the YAML hyper‑parameters.

Related Papers

The model’s scientific foundation is documented in the following open‑access paper:

• “SpeechBrain: A General‑Purpose Speech Toolkit” (arXiv:2106.04624) – the core SpeechBrain framework that provides the training recipes, loss functions, and inference utilities used for this emotion recognizer.

The IEMOCAP dataset itself is described in:

• Busso, C., et al. “IEMOCAP: Interactive Emotional Dyadic Motion Capture Database.” (2008) – the benchmark corpus on which the model is fine‑tuned.

These works together explain the choice of wav2vec2 as a pre‑trained acoustic encoder and the additive‑margin softmax loss that yields the high accuracy reported on the IEMOCAP test split.

Licensing Information

The repository lists the license as apache‑2.0 in the tags, but the official license field in the README is marked “unknown”. In practice, the underlying SpeechBrain code and the wav2vec2‑base weights are Apache‑2.0, which is a permissive open‑source license.

If you treat the model as Apache‑2.0, you may:

• Use the model for commercial or non‑commercial projects without paying royalties.
• Modify the code or fine‑tune the weights and redistribute the derived work.
• Include a copy of the Apache‑2.0 license and a notice of any changes you made.

Because the license field is ambiguous, it is prudent to:

• Check the Hugging Face model card for any updates on licensing.
• Contact the SpeechBrain maintainers if you plan to embed the model in a product that requires explicit licensing guarantees.