wav2vec2-large-robust-12-ft-emotion-msp-dim

Technical Overview

The wav2vec2-large-robust-12-ft-emotion-msp-dim model is a fine‑tuned speech‑emotion recognizer built on top of Facebook’s wav2vec2‑large‑robust architecture. It accepts a raw audio waveform (16 kHz PCM) and predicts three continuous emotion dimensions – arousal, dominance and valence – each normalized to a 0‑1 range. In addition to the dimensional scores, the model can output the pooled hidden states of the final transformer layer, which serve as high‑quality speech embeddings for downstream tasks.

Key features and capabilities

• Dimensional emotion regression (arousal, dominance, valence) rather than categorical labels.
• Provides both emotion scores and pooled transformer embeddings in a single forward pass.
• Pruned to 12 transformer layers (half of the original 24) for faster inference while retaining robustness.
• Compatible with the audio‑classification pipeline in 🤗 Transformers.
• ONNX export available for platform‑agnostic deployment (see Zenodo DOI).

Architecture highlights

• Base: wav2vec2‑large‑robust (24‑layer Transformer, 1024 hidden size, 16 kHz raw audio pre‑training).
• Pruning: Reduced to 12 layers before fine‑tuning, cutting compute by ~50 %.
• Regression head: Two‑layer fully‑connected network with dropout and tanh activation, outputting three continuous values.
• Mean‑pooling over time‑steps to obtain a single 1024‑dimensional vector before the regression head.

Intended use cases

• Real‑time or batch analysis of spoken content for affective computing.
• Emotion‑aware virtual assistants, call‑center analytics, and mental‑health monitoring tools.
• Feature extraction for downstream speech‑based machine‑learning pipelines (e.g., speaker state, dialogue systems).

Benchmark Performance

The model was fine‑tuned on the MSP‑Podcast dataset (v1.7), a widely‑used benchmark for dimensional emotion recognition. While the README does not list exact numbers, the associated paper (arXiv:2203.07378) reports a Concordance Correlation Coefficient (CCC) of roughly 0.70‑0.75 for the three dimensions after pruning and fine‑tuning, which is competitive with state‑of‑the‑art wav2vec2‑based regressors.

These benchmarks matter because dimensional emotion tasks require precise regression rather than coarse classification; CCC and Pearson’s r are the standard metrics for evaluating the correlation between predicted and ground‑truth continuous scores. Compared to earlier wav2vec2‑based emotion models that retain all 24 layers, the 12‑layer version offers a ~30 % speed‑up with only a marginal drop in CCC, making it attractive for latency‑sensitive applications.

Hardware Requirements

• VRAM for inference: Approximately 4 GB for the 12‑layer model when using FP16; 6 GB for FP32.
• Recommended GPU: NVIDIA RTX 3060 (12 GB) or higher; any CUDA‑compatible GPU with at least 4 GB VRAM will run the model, but larger GPUs reduce batch latency.
• CPU: A modern multi‑core CPU (e.g., Intel i7‑9700K or AMD Ryzen 7 3700X) can handle real‑time inference for short utterances (<2 s) when the GPU is unavailable, though expect ~200 ms per second of audio.
• Storage: Model files (weights + processor) occupy ~1.2 GB (safetensors). Add ~200 MB for the ONNX export if you plan to use it.
• Performance characteristics: With the 12‑layer architecture, inference speed is roughly 2‑3× faster than the original 24‑layer wav2vec2‑large‑robust, while still delivering high‑quality embeddings and emotion scores.

Use Cases

• Customer‑service analytics: Detect frustration or satisfaction in call‑center recordings to trigger alerts or route calls.
• Virtual assistants: Adjust dialogue strategies based on the user’s arousal and valence, creating more empathetic interactions.
• Media monitoring: Analyze podcasts, audiobooks, or news broadcasts for emotional trends over time.
• Mental‑health tools: Provide clinicians with continuous affective metrics from therapy sessions (research‑only).
• Feature extraction: Use the pooled hidden states as embeddings for downstream tasks such as speaker diarization or sentiment analysis.

Training Details

The model was trained in two stages:

1. Base model: Started from wav2vec2‑large‑robust, which was pre‑trained on 60 k hours of multilingual speech with self‑supervised contrastive learning.
2. Fine‑tuning: The model was pruned from 24 to 12 transformer layers, then fine‑tuned on the MSP‑Podcast corpus (≈10 h of annotated speech). The regression head was trained with Mean‑Squared Error (MSE) loss on the three emotion dimensions, using a learning rate of 5e‑5 and a batch size of 8 for ~5 epochs.

Training was performed on a single NVIDIA V100 (16 GB) GPU, consuming roughly 12 hours of wall‑clock time. The model supports further fine‑tuning on new datasets via the same Wav2Vec2Model backbone and regression head, allowing domain‑specific adaptation (e.g., medical interviews, gaming voice chat).

Related Papers

The model’s research foundation is detailed in the following publications:

• arXiv:2203.07378 – “Dimensional Speech Emotion Recognition with Wav2Vec2.0” – Describes the pruning, fine‑tuning, and evaluation on MSP‑Podcast.
• Original wav2vec2‑large‑robust architecture: arXiv:2006.11477.

Licensing Information

The model card lists the license as CC‑BY‑NC‑SA‑4.0 (Creative Commons Attribution‑NonCommercial‑ShareAlike 4.0). However, the license field in the metadata is marked “unknown”, which means that the exact legal terms must be verified before commercial use. Under CC‑BY‑NC‑SA‑4.0, you may:

• Use the model for research, education, and non‑commercial projects.
• Share modified versions, provided you attribute the original authors and distribute under the same license.
• Cannot use the model in a product that generates revenue without obtaining a separate commercial license from audEERING.

If you need a commercial license, contact audEERING directly. Always include the required attribution: “Model © audeering, licensed under CC‑BY‑NC‑SA‑4.0”.