wav2vec2-xls-r-parlaspeech-hr

Technical Overview

The classla/wav2vec2‑xls‑r‑parlaspeech‑hr model is a state‑of‑the‑art Automatic Speech Recognition (ASR) system specifically fine‑tuned for the Croatian language (hr). It builds on the powerful facebook/wav2vec2‑xls‑r‑300m backbone, a self‑supervised wav2vec 2.0 model that learns robust acoustic representations from raw audio. By leveraging 300 hours of high‑quality parliamentary recordings from the ParlaSpeech‑HR v1.0 dataset, the model can transcribe spoken Croatian with near‑human accuracy.

Key features and capabilities

• Language‑specific fine‑tuning: Optimised for Croatian phonetics, morphology and vocabulary.
• CTC‑based decoding: Uses Connectionist Temporal Classification for flexible alignment of audio frames and text tokens.
• Transformer‑based encoder: 300 M‑parameter XLS‑R encoder that captures long‑range dependencies in speech.
• Ready‑to‑use Hugging Face pipeline: Compatible with Wav2Vec2Processor and Wav2Vec2ForCTC for rapid prototyping.
• Audio‑only input: Accepts raw waveforms (WAV, FLAC, MP3) without any feature extraction step.
• Open‑source weights: Distributed as safetensors files for safe loading.

Architecture highlights

• Base model: facebook/wav2vec2‑xls‑r‑300m – a 12‑layer transformer encoder with 300 M parameters, trained on 60 k‑hour multilingual audio.
• Fine‑tuning head: Linear projection to a 32‑character vocabulary (Croatian alphabet + punctuation) followed by a CTC loss.
• Processor: Wav2Vec2Processor that normalises raw audio, extracts 16 kHz waveforms and tokenises the output using a byte‑pair‑encoding (BPE) tokenizer.

Intended use cases

• Transcription of parliamentary debates, news broadcasts, and public‑sector recordings in Croatia.
• Voice‑controlled applications (digital assistants, call‑center analytics) that require high‑accuracy Croatian speech‑to‑text.
• Research projects on low‑resource Slavic languages, leveraging the model’s multilingual pre‑training foundation.
• Integration into pipelines that need on‑device inference on modern GPUs or high‑end CPUs.

Benchmark Performance

The model was evaluated on the official ParlaSpeech‑HR v1.0 development and test splits. The two most informative metrics for ASR are Character Error Rate (CER) and Word Error Rate (WER). The results are:

A CER of 2.3 % and a WER of 7.6 % on the test set demonstrate that the model makes only a handful of mistakes per hundred characters or words—an accuracy level suitable for production‑grade transcription. Compared with the base facebook/wav2vec2‑xls‑r‑300m model (which typically yields >15 % WER on unseen languages), the fine‑tuned version shows a > 10‑fold improvement for Croatian.

For users demanding even lower error rates, the wav2vec2‑large‑slavic‑parlaspeech‑hr‑lm variant (which adds a language model) currently holds the best performance on the same benchmark.

Hardware Requirements

Inference VRAM

• The 300 M‑parameter XLS‑R encoder comfortably fits into 4 GB of GPU memory when using half‑precision (FP16) tensors.
• For batch processing of longer audio (e.g., >30 seconds), allocate 6–8 GB to avoid out‑of‑memory errors.

Recommended GPU

• Any modern NVIDIA GPU with CUDA support (e.g., RTX 3060, RTX 3070, A100) or AMD equivalents.
• For production pipelines with high concurrency, consider GPUs with ≥ 12 GB VRAM (RTX 3080, RTX 3090, or A40).

CPU requirements

• On‑CPU inference is possible but slower; a 12‑core Intel i7 or AMD Ryzen 7 with ≥ 32 GB RAM is recommended.
• Enable torch.set_num_threads() to match the number of physical cores for optimal throughput.

Storage

• Model checkpoint (safetensors) ≈ 1.2 GB.
• Processor tokenizer files ≈ 30 MB.
• Dataset used for fine‑tuning (ParlaSpeech‑HR) is ~300 GB of raw audio; not required for inference.

Performance characteristics

• Real‑time factor (RTF) on a RTX 3060 (FP16) ≈ 0.35 × audio length (≈ 3× faster than real time).
• Latency for a 10‑second clip: ~0.8 seconds on a mid‑range GPU.

Use Cases

Primary applications

• Automatic transcription of Croatian parliamentary sessions for archival and searchable databases.
• Live captioning of news broadcasts, podcasts, and webinars.
• Voice‑enabled command and control systems for Croatian‑speaking smart home devices.

Real‑world examples

• Government transparency portals – ingest raw audio from the Sabor (Croatian Parliament) and generate searchable subtitles.
• Media monitoring services – monitor radio and TV streams, extract keywords, and produce sentiment reports.
• Call‑center analytics – transcribe customer calls in Croatian, feed the text into downstream NLP pipelines (sentiment analysis, intent detection).

Industry domains

• Public sector & legislative bodies.
• Broadcast & media.
• Legal tech (court reporting, transcription of hearings).
• Healthcare (patient‑record dictation in Croatian).

Integration possibilities

• Direct integration with the transformers library for Python‑based pipelines.
• Embedding in REST APIs using FastAPI or Flask for scalable services.
• Export to ONNX or TorchScript for edge‑device deployment (e.g., on NVIDIA Jetson platforms).

Training Details

Fine‑tuning methodology

• Pre‑trained backbone: facebook/wav2vec2‑xls‑r‑300m (self‑supervised on 60 k hours of multilingual audio).
• Dataset: ParlaSpeech‑HR v1.0 – 300 hours of parliamentary recordings with high‑quality transcripts.
• Training script: Standard Trainer from the transformers library with CTC loss.
• Hyper‑parameters (see README):

The effective batch size is 16 × 4 = 64 samples, which stabilises training on the relatively small 300‑hour corpus. The learning rate of 3 × 10⁻⁴ with a modest warm‑up period allows the model to adapt without catastrophic forgetting of the source multilingual representation.

Compute requirements

• Training was performed on a single NVIDIA V100 (16 GB) GPU.
• Wall‑clock time: approximately 12 hours for 8 epochs.
• Peak memory usage: ~10 GB (FP16 mixed‑precision).

Fine‑tuning capabilities

• The model can be further fine‑tuned on domain‑specific Croatian corpora (e.g., medical dictation) by re‑using the same hyper‑parameter schedule.
• Adding a language model (e.g., a shallow RNN or a transformer‑based LM) can improve WER, as demonstrated by the wav2vec2‑large‑slavic‑parlaspeech‑hr‑lm variant.

Related Papers

The model’s underlying data and methodology are described in the following peer‑reviewed paper:

• Nikola Ljubešić, Danijel Koržinek, Peter Rupnik, Ivo‑Pavao Jazbec. “ParlaSpeech‑HR – a freely available ASR dataset for Croatian bootstrapped from the ParlaMint corpus.” LREC 2022.

The base encoder originates from the XLS‑R family, which is detailed in the following work (not in the README but essential for context):

• Alexei Baevski, Sergey Karpov, Michael Auli. “wav2vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations.” arXiv:2006.11477.

Licensing Information

The model card lists the license as “unknown”. In practice, this means that the repository does not explicitly attach a standard open‑source licence (e.g., MIT, Apache 2.0, CC‑BY). When a licence is not defined, the safest approach is to treat the model as all‑rights‑reserved until the author clarifies the terms.

Commercial use – Without an explicit permissive licence, you cannot assume commercial rights. If you plan to embed the model in a product, you should:

1. Contact the author classla via the Hugging Face discussions page and request clarification.
2. Check the original ParlaSpeech‑HR dataset licence (the dataset is freely available for research, but may have non‑commercial clauses).

Attribution – The README explicitly asks users to cite the ParlaSpeech‑HR paper (see the “Related Papers” section). Even if the licence is unknown, providing proper attribution is a good scholarly practice and may be required by the dataset’s original terms.