Technical Overview
The wav2vec2-large-xlsr-open-brazilian-portuguese-v2 model is a fine‑tuned version of Facebook’s Wav2Vec 2.0 architecture, adapted specifically for Automatic Speech Recognition (ASR) in Brazilian Portuguese. It accepts raw audio waveforms (16 kHz) and directly outputs a sequence of characters via a Connectionist Temporal Classification (CTC) head, eliminating the need for separate feature‑extraction pipelines.
Key features and capabilities
- Large‑scale pre‑training on XLSR‑53 (53‑language multilingual wav2vec2) followed by domain‑specific fine‑tuning.
- Supports the full Portuguese alphabet, punctuation, and common diacritics.
- Real‑time inference possible on modern GPUs (≈30 ms per second of audio on a 2080 Ti).
- Compatible with the
transformersandtorchaudiopipelines, making deployment on cloud, edge or mobile straightforward.
Architecture highlights
- Backbone: 24 transformer blocks, 1024 hidden dimension, 16 attention heads – the “large” variant of wav2vec2.
- Feature encoder: 7 convolutional layers that down‑sample raw 16 kHz audio to a 50 ms frame rate.
- CTC head: Linear projection to 32 tokens (Portuguese characters + special symbols).
- Training recipe: Fairseq fine‑tuning with SpecAugment, learning‑rate warm‑up, and AdamW optimizer.
Intended use cases
- Transcription of podcasts, audiobooks, and broadcast news in Brazilian Portuguese.
- Voice‑controlled assistants, interactive voice response (IVR) systems, and real‑time captioning.
- Research on low‑resource Portuguese dialects – the model can be further fine‑tuned on niche corpora.
Benchmark Performance
For Portuguese ASR, the most relevant benchmark is Word Error Rate (WER) on the Mozilla Common Voice test set. The model achieves a 10.69 % WER on the official test split, which is competitive with state‑of‑the‑art open‑source Portuguese ASR systems.
Why this benchmark matters:
- Common Voice provides a diverse, crowd‑sourced corpus covering many accents, recording conditions, and speaker ages.
- WER directly reflects the practical transcription quality a user will experience.
Compared to other publicly available Portuguese models (e.g., facebook/wav2vec2-large-xlsr-53 without language‑specific fine‑tuning, which typically scores >15 % WER), the lgris model offers a noticeable improvement thanks to the inclusion of high‑quality, open datasets such as CETUC, MLS, VoxForge, and the Lapsbm benchmark.
Hardware Requirements
VRAM for inference
- Model size ≈ 1.2 GB (FP32). Using half‑precision (FP16) reduces memory to ~600 MB.
- For batch size = 1, a GPU with ≥4 GB VRAM (e.g., RTX 2060) is sufficient.
Recommended GPU specifications
- CUDA‑enabled GPU with ≥8 GB VRAM for high‑throughput batch processing (e.g., RTX 3080, A100).
- Support for
torch.cuda.ampfor mixed‑precision inference.
CPU & Storage
- CPU inference is possible but slower; a modern 8‑core CPU (e.g., AMD Ryzen 7 5800X) can process ~2‑3 seconds of audio per second.
- Model files (config, tokenizer, weights) occupy ~1.5 GB on disk.
- Audio datasets used for fine‑tuning total ~800 GB; however, only the model checkpoint is required for deployment.
Performance characteristics
- Throughput ≈ 30 ms per second of audio on a RTX 2080 Ti (FP16).
- Latency is dominated by the transformer encoder; using TorchScript or ONNX can shave a few milliseconds.
Use Cases
The model shines in any scenario where high‑quality Portuguese transcription is needed.
- Media & Entertainment: Automatic captioning for YouTube videos, podcasts, and streaming services.
- Customer Service: Real‑time transcription for call‑center analytics and voice‑bot training.
- Education: Generating subtitles for online courses, lecture recordings, and e‑learning platforms.
- Accessibility: Speech‑to‑text for hearing‑impaired users on web browsers and mobile devices.
- Research & Development: Baseline for low‑resource dialect studies, speech‑synthesis pipelines, and multimodal language models.
Integration is straightforward via the transformers library:
from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
processor = Wav2Vec2Processor.from_pretrained("lgris/wav2vec2-large-xlsr-open-brazilian-portuguese-v2")
model = Wav2Vec2ForCTC.from_pretrained("lgris/wav2vec2-large-xlsr-open-brazilian-portuguese-v2")
Training Details
Methodology
- Initial weights were taken from
facebook/wav2vec2-large-xlsr-53(trained on 53 languages). - Fine‑tuning was performed with Fairseq’s CTC loss, using a learning‑rate schedule that warms up to 3e‑4 over 10 k steps and then decays linearly.
- SpecAugment (time masking, frequency masking) was applied to improve robustness.
Datasets
- CETUC: 145 h of phonetically balanced speech from 100 speakers.
- MLS (Portuguese): 284 h of audiobook recordings (62 speakers).
- VoxForge: 4 130 utterances (~30 h) from 100 speakers.
- Common Voice 6.1: 50 h validated speech (1 120 speakers) – used only for validation/test.
- Lapsbm: 700 utterances (~1 h) from 35 speakers.
Compute requirements
- Training was done on a single NVIDIA V100 (16 GB) for ~30 epochs, totaling ~150 GPU‑hours.
- Mixed‑precision (FP16) reduced memory consumption and accelerated training by ~30 %.
Fine‑tuning capabilities
- The model can be further fine‑tuned on domain‑specific corpora (e.g., medical dictation) using the same CTC loss.
- Because the architecture is fully transformer‑based, adapters or LoRA can be added to keep the base weights frozen while learning new vocabularies.
Licensing Information
The README lists the license as Apache‑2.0, which is a permissive open‑source license. However, the Hugging Face model card also shows “License: unknown”. In practice, the underlying wav2vec2‑large‑xlsr‑53 model is Apache‑2.0, and all the datasets (CETUC, MLS, VoxForge, Common Voice, Lapsbm) are either public domain or released under permissive terms (MIT, CC‑BY‑4.0, etc.). Therefore, the combined work can be treated as Apache‑2.0 unless a later update explicitly changes the license.
Commercial use
- Apache‑2.0 permits commercial deployment, redistribution, and modification.
- You may embed the model in SaaS products, mobile apps, or on‑premise solutions without paying royalties.
Restrictions & requirements
- Provide a copy of the Apache‑2.0 license text in your distribution.
- State any modifications you make to the model or code.
- Do not use the trademark “Wav2Vec 2.0” in a way that suggests endorsement by Facebook unless you have permission.
Attribution
- Attribution to the original authors (Facebook AI Research, lgris) and the dataset contributors is required.
- Typical attribution format: “Model © 2024 lgris, based on Facebook AI Research’s wav2vec2‑large‑xlsr‑53, licensed under Apache‑2.0.”