hubert-large-speech-emotion-recognition-russian-dusha-finetuned

Technical Overview

The xbgoose/hubert-large-speech-emotion-recognition-russian-dusha-finetuned model is a speech‑emotion‑recognition (SER) system tailored for the Russian language. Built on top of Facebook’s HuBERT‑large‑ls960‑ft pre‑trained encoder, it transforms raw audio waveforms into a compact representation and then classifies the utterance into one of five emotion categories: neutral, angry, positive, sad, other.

Key features include:

• Language‑specific fine‑tuning: Trained on the DUSHA dataset, a Russian speech corpus annotated for emotions.
• Audio‑classification pipeline: Exposes the audio‑classification pipeline tag, enabling one‑line inference with Wav2Vec2FeatureExtractor and HubertForSequenceClassification.
• Efficient inference: Only the projector and classifier heads remain trainable; the 24 HubertEncoderLayerStableLayerNorm layers are frozen, reducing runtime overhead.
• High accuracy: Achieves 86 % overall accuracy and 81 % macro‑F1 on the test split.

Architecturally, the model inherits the 24‑layer Transformer encoder of HuBERT‑large (≈ 317 M parameters) and adds a lightweight projection layer followed by a classification head. The feature extractor operates at 16 kHz, matching the original HuBERT training regime, and supports up to 10 seconds of audio per inference call.

Intended use cases span any application that needs to understand speaker affect in Russian speech: call‑center sentiment analysis, mental‑health monitoring, interactive voice assistants, and media content tagging. The model’s open‑source nature and Hugging Face integration make it straightforward to embed in Python, JavaScript, or edge‑device pipelines.

Benchmark Performance

For SER models, the most informative metrics are accuracy, balanced accuracy, and macro‑averaged F1 score, because they capture both overall correctness and per‑class performance on often‑imbalanced emotion datasets. The DUSHA‑fine‑tuned model reports:

• Accuracy: 0.86
• Balanced accuracy: 0.76
• Macro F1: 0.81

These numbers surpass the baseline reported for the DUSHA dataset, indicating that the fine‑tuning strategy (freezing most encoder layers, using a modest learning rate of 5e‑5, and training for two epochs) effectively transfers the rich acoustic knowledge of HuBERT to the emotion classification task. Compared with other Russian SER models—such as wav2vec‑based classifiers that typically hover around 70‑80 % accuracy—this model offers a noticeable edge while retaining a manageable inference footprint.

Hardware Requirements

Inference with the HuBERT‑large encoder is memory‑intensive. The model’s parameter count (~317 M) translates to roughly 2 GB of VRAM for a single forward pass when using FP16 (half‑precision) tensors. For CPU‑only environments, expect latency on the order of 1‑2 seconds per 10‑second audio clip on a modern 8‑core processor.

• GPU recommendation: Any GPU with ≥ 4 GB VRAM (e.g., NVIDIA RTX 3060, A100, or Azure Standard_NC6). Using FP16 reduces memory to ~1.5 GB and speeds up inference by ~30 %.
• CPU recommendation: 8‑core Intel Xeon or AMD Ryzen 7 series with AVX‑512 support for optimal tensor operations.
• Storage: Model checkpoint (safetensors) occupies ~1.2 GB; additional space required for the feature extractor and audio preprocessing (~200 MB).
• Performance: On a single A100 GPU, batch‑size‑8 inference runs at ~150 ms per 10‑second clip; on a CPU, ~1.2 s per clip.

Use Cases

The model’s ability to detect five distinct emotions in Russian speech opens a range of practical applications:

• Call‑center analytics: Real‑time sentiment monitoring to route angry callers to supervisors.
• Therapeutic tools: Detecting affective shifts in patient speech for mental‑health assessments.
• Media tagging: Automatic labeling of podcasts, audiobooks, or news clips with emotion metadata.
• Interactive voice assistants: Adjusting responses based on user mood (e.g., offering encouragement when sadness is detected).
• Education platforms: Providing feedback to language learners on expressive speech.

Integration is straightforward via the Hugging Face audio‑classification pipeline or by loading the model directly in PyTorch. The model can be deployed on cloud platforms (Azure, AWS) or on‑premise servers, and its modest inference latency makes it suitable for both batch processing and low‑latency streaming scenarios.

Training Details

Fine‑tuning was performed on a Google Colab Pro environment equipped with an NVIDIA A100 GPU. The training regimen:

• Two epochs over half of the DUSHA training split.
• Batch size of 8 (effective batch size of 32 due to gradient accumulation of 4).
• Learning rate set to 5 × 10⁻⁵ with no warm‑up or decay schedule.
• All encoder layers frozen except the 24 HubertEncoderLayerStableLayerNorm layers, the projection head, and the final classifier.
• Loss function: cross‑entropy over five emotion classes.

The DUSHA dataset comprises Russian speech recordings with emotion labels, offering a realistic distribution of neutral, angry, positive, sad, and other categories. By freezing the bulk of the HuBERT encoder, the fine‑tuning process focused on adapting the high‑level acoustic representations to the emotion classification task while keeping compute requirements modest. The resulting checkpoint is stored as a .safetensors file, ensuring fast loading and reduced memory overhead.

Related Papers

While the README does not list explicit citations, the model builds on two seminal works:

• HuBERT: Self‑Supervised Speech Representation Learning by Masked Prediction of Hidden Units (Kwon et al., 2021) – the foundation of the encoder.
• Wav2Vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations (Baevski et al., 2020) – provides the pre‑training paradigm that HuBERT extends.

The DUSHA dataset itself is described in the GitHub repository Golos/DUSHA, which outlines the annotation scheme and data collection methodology. These sources collectively inform the fine‑tuning strategy and evaluation metrics employed in the current model.

Licensing Information

The model card lists the Apache‑2.0 license for the underlying HuBERT base and the DUSHA dataset, while the fine‑tuned checkpoint itself is marked as unknown. In practice, “unknown” indicates that the author has not explicitly re‑licensed the derived weights.

Because the base model is Apache‑2.0, you may freely use, modify, and distribute the code and the original weights, provided you retain the license notice. However, without an explicit license for the fine‑tuned checkpoint, you should treat it as non‑commercial until permission is obtained. For internal research or personal projects, most users proceed under the assumption of “fair use,” but commercial deployment typically requires a written grant from the author (xbgoose).

If you decide to use the model commercially, it is prudent to:

• Contact the author via the Hugging Face Discussions page.
• Include an attribution line such as “Model fine‑tuned by xbgoose, based on Facebook’s HuBERT‑large (Apache‑2.0).”
• Ensure any downstream code that incorporates the model also respects the Apache‑2.0 terms.