Technical Overview
Model ID: jonatasgrosman/wav2vec2-large-xlsr-53-portuguese
Model Name: wav2vec2-large-xlsr-53-portuguese
Author: Jonatas Grosman
This model is a fine‑tuned version of Facebook’s wav2vec2‑large‑xlsr‑53 specifically adapted for Portuguese speech recognition. It converts raw audio sampled at 16 kHz into text using a Connectionist Temporal Classification (CTC) head. The base wav2vec2‑large‑xlsr‑53 architecture is a self‑supervised transformer that learns powerful acoustic representations from 53 k of multilingual audio, and the fine‑tuning step aligns those representations to Portuguese phonetics and orthography.
Key Features & Capabilities
- Supports end‑to‑end Automatic Speech Recognition (ASR) for European Portuguese (pt‑PT) and Brazilian Portuguese (pt‑BR).
- Works out‑of‑the‑box with the Huggingsound library for rapid prototyping.
- Compatible with 🤗 Transformers and datasets for custom pipelines.
- Provides both raw CTC output and optional language‑model (LM) rescoring, achieving a 9.01 % WER with a 3‑gram LM on the Common Voice test set.
- Optimized for 16 kHz mono audio, the most common sampling rate for telephony and voice‑assistant recordings.
Architecture Highlights
- Transformer encoder with 24 layers, 1024 hidden size, and 16 attention heads (≈300 M parameters).
- Self‑supervised pre‑training on 53 k of multilingual audio (XLS‑R) gives robust cross‑lingual acoustic features.
- CTC classification head maps encoder outputs to a Portuguese‑specific token set (including letters, punctuation, and special symbols).
- Fine‑tuned on the Common Voice 6.0 Portuguese split, leveraging ~200 h of high‑quality speech.
Intended Use Cases
- Voice assistants and chatbots that need Portuguese language support.
- Transcription services for podcasts, interviews, and call‑center recordings.
- Accessibility tools such as real‑time captioning for the deaf and hard‑of‑hearing.
- Research projects that explore low‑resource language ASR or multilingual transfer learning.
Benchmark Performance
The model’s performance is reported on two benchmark suites that matter for Portuguese ASR:
- Common Voice pt (test set) – Word Error Rate (WER) = 11.31 % and Character Error Rate (CER) = 3.74 % without a language model. With a 3‑gram LM, WER drops to 9.01 % and CER to 3.21 %.
- Robust Speech Event – Dev Data – A more challenging noisy‑environment set where the model achieves WER = 42.1 % (36.92 % with LM) and CER = 17.93 % (16.88 % with LM).
These metrics are crucial because they reflect both clean‑speech conditions (Common Voice) and real‑world noisy scenarios (Robust Speech Event). Compared to other Portuguese ASR models on the 🤗 ASR leaderboard, the 9 % WER with LM places this model among the top‑performing open‑source solutions, especially given its modest compute footprint for inference.
Hardware Requirements
Running wav2vec2‑large‑xlsr‑53‑portuguese efficiently requires a GPU with sufficient VRAM for the 300 M‑parameter transformer. Below are practical guidelines:
- VRAM for inference: 8 GB is the minimum for batch size = 1; 12 GB+ allows larger batches or longer audio chunks without truncation.
- Recommended GPUs: NVIDIA RTX 3060 (12 GB), RTX 3070 (8 GB), RTX 3080 (10‑12 GB), or any recent AMD Radeon with comparable memory.
- CPU: A modern multi‑core CPU (e.g., Intel i7‑10700K or AMD Ryzen 7 3700X) can handle preprocessing and feeding the model, but GPU inference dominates performance.
- Storage: The model checkpoint is ~1.2 GB; include the model files and a small cache for tokenizers (≈50 MB).
- Performance: On a RTX 3060, a 10‑second audio clip is transcribed in ~0.6 seconds (real‑time factor ≈ 0.06). Larger batches improve throughput proportionally.
Use Cases
The model’s strong performance on both clean and noisy Portuguese speech makes it suitable for a wide range of applications:
- Customer‑service analytics: Automatic transcription of call‑center recordings for sentiment analysis and quality monitoring.
- Media monitoring: Real‑time captioning of live broadcasts, podcasts, and webinars in Portuguese.
- Assistive technology: Speech‑to‑text for people with disabilities, integrated into mobile or desktop accessibility suites.
- Educational platforms: Automatic grading of spoken language exercises and language‑learning apps.
- Research & prototyping: Fast baseline for academic studies on low‑resource ASR, domain adaptation, or multilingual transfer learning.
Training Details
The model was fine‑tuned on the Portuguese split of Common Voice 6.0, using the train and validation subsets. The training pipeline is publicly available in the wav2vec2‑sprint repository. Key aspects of the training process include:
- Pre‑training base:
facebook/wav2vec2-large-xlsr-53(≈300 M parameters). - Fine‑tuning data: ~200 hours of Portuguese speech with corresponding transcripts.
- Optimization: AdamW optimizer, learning rate 3e‑5, linear warm‑up for 10 % of steps, and CTC loss.
- Compute: Trained on OVHcloud GPU credits (likely a V100/A100 instance); total training time ~12 hours for 30 k steps.
- Language‑model rescoring: A 3‑gram LM built from the Common Voice transcripts improves WER by ~2 % absolute.
- Fine‑tuning flexibility: Users can continue training on domain‑specific data (e.g., medical or legal Portuguese) by loading the checkpoint with
Wav2Vec2ForCTCand following the same script.
Licensing Information
The model is released under the Apache 2.0 license (as indicated in the README). This permissive license grants the following rights:
- Free use, modification, and distribution for both personal and commercial projects.
- Ability to embed the model in proprietary software without open‑sourcing the downstream code.
- Obligation to retain the original copyright notice and provide a copy of the license in any redistribution.
- No warranty; the model is provided “as‑is”.
If you distribute a product that incorporates the model, you must include the Apache 2.0 license text and attribute the original author (Jonatas Grosman). There are no “unknown” restrictions beyond the standard Apache clauses.