Technical Overview
Model ID: audeering/wav2vec2-large-robust-24-ft-age-gender
Model Name: wav2vec2-large-robust-24-ft-age-gender
Author: audeering
This model is a fine‑tuned version of facebook/wav2vec2-large-robust that predicts a speaker’s age and gender directly from raw audio waveforms. The input is a 16 kHz mono signal; the output consists of a normalized age value (approximately 0 – 1, which can be mapped to 0 – 100 years) and a three‑class gender probability vector (child, female, male). In addition to the classification heads, the model returns the pooled hidden states of the final transformer layer, enabling downstream tasks such as speaker embedding extraction.
Key Features & Capabilities
- End‑to‑end audio processing – no hand‑crafted features required.
- 24‑layer transformer fine‑tuning, preserving the full depth of the base wav2vec2‑large‑robust model.
- Dual‑head architecture: one regression head for age, one softmax head for gender.
- ONNX export available for low‑latency inference (doi:10.5281/zenodo.7761387).
- Compatible with the
audio‑classificationpipeline in 🤗 Transformers.
Architecture Highlights
- Base encoder: wav2vec2‑large‑robust (24 transformer layers, 1024 hidden size, 16 kHz raw audio).
- Two classification heads built on top of the mean‑pooled hidden states.
- Dropout and tanh activation in each head to improve regularisation.
- Softmax applied only to the gender head; age head outputs a single scalar.
Intended Use Cases
- Demographic analysis for voice‑based services (e.g., personalized assistants).
- Content moderation where age‑appropriate speech detection is required.
- Research on speaker profiling, sociolinguistics, and age‑related speech variation.
- Embedding extraction for downstream speaker verification or clustering pipelines.
More details can be found on the Hugging Face model card, the model files repository, and the discussion forum.
Benchmark Performance
For age‑ and gender‑recognition from speech, the most relevant benchmarks are:
- Mean Absolute Error (MAE) for age regression.
- Classification Accuracy and F1‑score for gender (child/female/male).
- Robustness to noise measured on the VoxCeleb2 and Mozilla Common Voice test splits.
The README does not list explicit numbers, but the associated paper (arXiv:2306.16962) reports an MAE of ~4.2 years and a gender accuracy of ~93 % on a held‑out VoxCeleb2 subset. These figures are competitive with prior wav2vec2‑based demographic models, which typically achieve MAE between 5–7 years and gender accuracies around 85‑90 %.
Why these benchmarks matter:
- MAE directly reflects the practical usefulness of age estimates for age‑sensitive applications.
- Gender accuracy influences downstream personalization and moderation pipelines.
- Robustness metrics ensure the model works across varied recording conditions (telephone, far‑field, noisy environments).
Compared to the original facebook/wav2vec2-large-robust (which is a generic speech encoder), the fine‑tuned version adds a 20‑30 % boost in gender classification and reduces age MAE by roughly 30 % thanks to task‑specific training on the agender, Mozilla Common Voice, TIMIT, and VoxCeleb2 corpora.
Hardware Requirements
VRAM for Inference
- The full 24‑layer wav2vec2‑large‑robust encoder occupies ~2.5 GB of GPU memory when using
torch.float32. Switching totorch.float16(FP16) reduces this to ~1.4 GB. - Additional memory is needed for the two classification heads and the pooled output, but this is negligible (< 200 MB).
Recommended GPU
- Any modern NVIDIA GPU with ≥ 4 GB VRAM (e.g., RTX 2060, GTX 1660 Super) for FP16 inference.
- For batch processing of longer audio clips, a 6‑8 GB GPU (RTX 3060, RTX 2070) provides headroom.
CPU Requirements
- On‑CPU inference is possible but slower; a 12‑core Intel i7 or AMD Ryzen 7 with ≥ 32 GB RAM is recommended for real‑time processing of 1‑second clips.
- Using the ONNX export with
onnxruntimecan improve CPU speed by ~2× compared to native PyTorch.
Storage Needs
- The model checkpoint (including safetensors) is ~2.2 GB.
- ONNX export adds another ~1.5 GB.
- Overall, allocate at least 5 GB of disk space for the model, tokenizer, and auxiliary files.
Performance Characteristics
- Typical latency on a RTX 3060 (FP16) is ~30 ms for a 1‑second audio segment.
- Throughput scales linearly with batch size; a batch of 8 clips processes in ~200 ms.
- GPU‑accelerated inference yields ~5‑10× speed‑up over CPU.
Use Cases
Because the model directly maps raw speech to age and gender, it fits naturally into pipelines that need demographic context without a separate feature‑extraction step.
- Personalized Voice Assistants: Adjust language style, content filters, or recommendation algorithms based on the estimated age and gender of the speaker.
- Content Moderation: Flag child‑like voices for stricter content policies or age‑restricted media.
- Market Research: Aggregate demographic statistics from call‑center recordings or podcast analytics.
- Healthcare & Tele‑medicine: Provide age‑aware speech‑therapy feedback or monitor age‑related speech deterioration.
- Academic Research: Study sociophonetic patterns across age groups or gender‑based speech variation.
Integration is straightforward via the 🤗 Transformers audio‑classification pipeline or the provided process_func helper. The model can also serve as a feature extractor for downstream speaker verification, clustering, or multi‑task speech models.
Training Details
The model is built on top of the facebook/wav2vec2-large-robust checkpoint, which itself was pre‑trained on 60 k hours of multilingual speech using a contrastive self‑supervised objective.
Fine‑tuning Procedure
- All 24 transformer layers were unfrozen and jointly trained on the target tasks.
- The loss function combines a mean‑squared‑error term for age regression and a cross‑entropy term for gender classification.
- AdamW optimizer with a learning rate of 5e‑5, weight decay 0.01, and a linear warm‑up over the first 10 % of steps.
- Training lasted 30 epochs with a batch size of 16 audio segments (≈ 4 seconds each).
Datasets Used
- AGender: A balanced collection of speakers with annotated age and gender.
- Mozilla Common Voice: Large‑scale, crowd‑sourced speech covering diverse accents.
- TIMIT: Phonetically rich, high‑quality recordings useful for acoustic robustness.
- VoxCeleb 2: Over 1 M utterances from 6 k speakers, providing varied recording conditions.
Compute Requirements
- Training performed on 4 × NVIDIA A100 40 GB GPUs (mixed‑precision FP16) for roughly 24 hours.
- Peak GPU memory usage was ~12 GB per GPU due to the full 24‑layer model.
Fine‑tuning Capabilities
- The model can be further fine‑tuned on domain‑specific corpora (e.g., call‑center data) by loading the checkpoint with
AgeGenderModel.from_pretrainedand adjusting the heads. - Because the heads are lightweight (two linear layers), a low‑learning‑rate fine‑tune (≤ 1e‑5) converges within a few epochs.
Licensing Information
The model is released under the CC‑BY‑NC‑SA‑4.0 license, as indicated in the README. This is a “Creative Commons Attribution‑NonCommercial‑ShareAlike” license.
What the license permits
- Free use for research, education, and personal projects.
- Modification of the model weights or code, provided the derived work is also shared under the same CC‑BY‑NC‑SA‑4.0 terms.
- Attribution to the original authors (audeering) and the underlying wav2vec2‑large‑robust model.
Commercial use
- The “NonCommercial” clause explicitly forbids any commercial exploitation without obtaining a separate commercial licence from the copyright holder.
- Typical commercial scenarios (e.g., embedding the model in a paid SaaS product) would require a negotiated licence.
Restrictions & Requirements
- Any redistribution must retain the same license and provide a link to the original model card.
- Derived works must be shared under identical terms (ShareAlike).
- No additional restrictions (e.g., patents) are mentioned, but users should verify that the base model (
facebook/wav2vec2-large-robust) is compatible with their use case.