wav2vec2-large-xlsr-catala

Technical Overview

The softcatala/wav2vec2-large-xlsr-catala model is a Catalan‑specific automatic‑speech‑recognition (ASR) system built on top of Facebook’s wav2vec 2.0 Large XLSR‑53 architecture. It transforms raw audio waveforms sampled at 16 kHz into Catalan text without requiring an external language model. The model has been fine‑tuned on two high‑quality Catalan corpora – Common Voice (ca) and ParlamentParla – to capture both everyday speech and formal parliamentary language.

Key features & capabilities

• End‑to‑end speech‑to‑text with CTC decoding.
• Supports 16 kHz mono audio; resampling from higher rates is built‑in in the example code.
• High‑accuracy Catalan transcription (WER ≈ 7 % on a mixed test set).
• Ready‑to‑use with the Wav2Vec2Processor and Wav2Vec2ForCTC classes from the 🤗 Transformers library.
• Fully compatible with PyTorch, TensorFlow (via JAX), and deployment platforms such as Azure.

Architecture highlights

• Base model: facebook/wav2vec2-large-xlsr-53 – 300 M parameters, 24‑layer Transformer encoder, 12 M‑size convolutional feature extractor.
• Fine‑tuning adds a linear CTC head on top of the encoder to map hidden representations to the Catalan character set.
• Training leveraged the “XLSR‑fine‑tuning‑week” recipe, which freezes the low‑level feature extractor for the first few epochs and then unfreezes the entire network.

Intended use cases

• Live captioning of Catalan broadcasts, podcasts, and webinars.
• Transcription of parliamentary debates and public‑sector meetings.
• Voice‑controlled Catalan applications (virtual assistants, smart home devices).
• Data‑annotation pipelines for building Catalan language resources.

Benchmark Performance

Benchmarking for ASR models focuses on Word Error Rate (WER), which measures the proportion of insertions, deletions, and substitutions relative to a reference transcript. Lower WER indicates more accurate transcription.

These numbers are computed on a held‑out test.csv that the model never saw during training, ensuring an unbiased estimate. Compared with multilingual wav2vec 2.0 baselines, the Catalan‑fine‑tuned model achieves a ~5‑7 % absolute WER reduction on in‑domain data, making it competitive with commercial Catalan ASR services.

Hardware Requirements

Inference VRAM

• The large XLSR‑53 encoder occupies ~1.2 GB of GPU memory for a single‑utterance forward pass (FP32). Using half‑precision (FP16) reduces this to ~650 MB.
• For batch processing (e.g., 8‑second clips in a batch of 8), a GPU with at least 4 GB VRAM is recommended.

Recommended GPU

• Desktop: NVIDIA RTX 3060 (12 GB) or RTX 3070 (8 GB) – ample headroom for real‑time streaming.
• Edge: NVIDIA Jetson AGX Xavier (16 GB) – can run the model with FP16 quantisation.

CPU & Storage

• CPU‑only inference is possible but slower; a modern 8‑core CPU (e.g., AMD Ryzen 7 5800X) can process ~1‑second audio in ~300 ms.
• Model files (weights + processor) total ~1.5 GB; keep at least 2 GB free on the storage device.

Performance characteristics

• Real‑time factor (RTF) ≈ 0.3 on a RTX 3060 (i.e., 1 second of audio processed in ~0.3 seconds).
• Latency dominated by audio loading and resampling; using torchaudio’s native resampler keeps overhead < 10 ms.

Use Cases

Primary applications

• Live transcription of Catalan news broadcasts and podcasts.
• Automatic captioning for e‑learning platforms serving Catalan‑speaking students.
• Voice‑controlled smart‑home assistants that understand Catalan commands.
• Speech analytics for Catalan parliamentary sessions and public‑sector meetings.

Real‑world examples

• Media company “TV3” could integrate the model to generate subtitles for live programmes.
• University of Barcelona’s language‑lab could use it to transcribe student oral exams.
• Government portals could offer searchable audio archives of council meetings.

Integration possibilities

• Deploy via 🤗 Transformers pipeline("automatic-speech-recognition") in a Flask or FastAPI service.
• Export to ONNX for low‑latency inference on edge devices.
• Combine with a Catalan language model (e.g., KenLM) for higher‑order decoding.

Training Details

Methodology

• Base model: facebook/wav2vec2-large-xlsr-53 (self‑supervised pre‑training on 53 languages).
• Fine‑tuning performed with a CTC loss on Catalan transcriptions.
• Training script from the ccoreilly/wav2vec2-catala repo, using the “XLSR‑fine‑tuning‑week” schedule (freeze feature extractor for 2 epochs, then unfreeze).

Datasets

• Common Voice (ca) – crowd‑sourced Catalan speech, diverse speakers, varied acoustic conditions.
• ParlamentParla – high‑quality recordings of Catalan parliamentary debates, formal diction and longer utterances.
• Both datasets were combined into a custom train/dev/test split (see test.csv).

Compute requirements

• Training performed on a single NVIDIA V100 (16 GB) for ~1 week (≈ 120 h) using a batch size of 8 seconds of audio.
• Mixed‑precision (FP16) training reduced memory usage and accelerated convergence.

Fine‑tuning capabilities

• The model can be further fine‑tuned on domain‑specific Catalan corpora (e.g., medical or legal speech) using the same CTC pipeline.
• Because the base encoder is frozen initially, additional fine‑tuning requires only a modest GPU (8 GB) and a few epochs to adapt to new vocabularies.

Related Papers

The underlying technology is described in the original wav2vec 2.0 paper:

• “wav2vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations” – Baevski et al., 2020.
• For the multilingual XLSR‑53 pre‑training, see “Cross‑Lingual Speech Representation Learning” – Conneau et al., 2020.

The fine‑tuning methodology follows the “XLSR‑fine‑tuning‑week” recipe, which is documented in the Hugging Face audio examples repository.

Licensing Information

The repository tags the model with license:apache-2.0, but the top‑level card lists the license as “unknown”. In practice, the underlying facebook/wav2vec2-large-xlsr-53 model is released under the Apache 2.0 licence, and the fine‑tuning scripts and data usage are also Apache‑compatible.

What Apache 2.0 allows

• Free use, modification, and distribution – both personal and commercial.
• Patents granted by contributors are covered, reducing legal risk.
• Requires preservation of the original copyright notice and a copy of the licence in redistributed binaries.

Potential restrictions

• If the “unknown” status persists, double‑check the GitHub licence file (if present) before commercial deployment.
• Some downstream datasets (e.g., Common Voice) have their own attribution requirements; ensure compliance when using the model for commercial products.