wav2vec2-xls-r-300m-hebrew

Technical Overview

The wav2vec2‑xls‑r‑300m‑hebrew model (ID imvladikon/wav2vec2-xls-r-300m-hebrew) is a Hebrew‑focused automatic speech recognition (ASR) system. It is built on top of Facebook’s wav2vec2‑xls‑r‑300m foundation, a 300‑million‑parameter self‑supervised speech encoder that has been pre‑trained on 53 khours of multilingual audio. The model has been fine‑tuned in two stages on a private Hebrew corpus, first on a curated high‑quality set (≈28 h) and then on a larger, more diverse collection (≈69 h) that also contains weakly‑labeled data.

Key features & capabilities

• Supports the he language code – native Hebrew transcription.
• End‑to‑end ASR pipeline (automatic‑speech‑recognition tag) ready for 🤗 Transformers pipelines.
• Optimized for both CPU and GPU inference with torch and safetensors weights.
• Robust to varied acoustic conditions thanks to the “robust‑speech‑event” tag and the mixed‑source training data.

Architecture highlights

• Base encoder: facebook/wav2vec2-xls-r-300m – 12 transformer blocks, 300 M parameters.
• Fine‑tuned classification head for CTC (Connectionist Temporal Classification) decoding.
• Training employed native AMP mixed‑precision, linear learning‑rate schedule with 1 000 warm‑up steps, and a total batch size of 64 (effective) across 2 GPUs.

Intended use cases

• Live or batch transcription of Hebrew audio (podcasts, lectures, call‑center recordings).
• Voice‑controlled applications for Hebrew‑speaking users.
• Pre‑processing for downstream NLP tasks such as sentiment analysis or keyword spotting.

Benchmark Performance

The most relevant benchmark for an ASR model is Word Error Rate (WER). The README reports a final test WER of 23.18 % on a custom Hebrew test set after the second fine‑tuning stage. Earlier checkpoints achieved 17.7 % WER on the small high‑quality set and 23.2 % on the larger, more diverse set, demonstrating consistent performance across data domains.

WER is crucial because it directly reflects transcription accuracy as perceived by end‑users. A sub‑25 % WER is competitive for Hebrew, a language with relatively limited public ASR resources, and places this model ahead of many generic multilingual wav2vec2‑based baselines that typically hover above 30 % WER on Hebrew.

Hardware Requirements

The underlying wav2vec2‑xls‑r‑300m encoder requires roughly 2 GB–3 GB of VRAM for a single‑utterance inference pass when using the safetensors format and torch‑scripted model. For batch inference (e.g., 8‑sample batches) a GPU with at least 6 GB VRAM is recommended to keep latency low.

• GPU: NVIDIA RTX 3060 or higher (CUDA 11+, cuDNN 8+).
• CPU: Modern x86_64 with AVX2 support; 8 cores provide decent throughput for CPU‑only inference.
• Storage: Model files (weights + config) occupy ~1.2 GB; keep an additional 2 GB free for temporary tensors.
• Performance: On a RTX 3070, real‑time transcription of 16 kHz audio is achievable at ~30 ms per second of audio.

Use Cases

The model shines in any scenario that requires accurate Hebrew speech transcription:

• Media & Entertainment: Automatic subtitles for Hebrew TV shows, movies, or YouTube videos.
• Education: Transcribing lectures, language‑learning podcasts, and oral exams.
• Customer Support: Real‑time call‑center transcription for quality monitoring and analytics.
• Accessibility: Voice‑to‑text tools for hearing‑impaired users.
• Voice‑Driven Apps: Smart‑home assistants or mobile dictation apps targeting Hebrew‑speaking markets.

Training Details

Fine‑tuning was performed in two distinct stages:

• Stage 1 (small high‑quality set): 4.19 GB of audio (20 306 samples, 28 h) + 1.05 GB dev set (5 076 samples, 7 h). The model reached a validation WER of 0.1773 (≈17.7 %).
• Stage 2 (large mixed set): 12.3 GB training data (90 777 samples, 69 h) plus a 2.39 GB dev set (20 246 samples, 14 h) that excluded weakly‑labeled data. After this stage the test WER improved to 0.2318 (≈23.2 %).

Both stages used the same hyper‑parameters: learning rate 3e‑4, batch size 8 per GPU, gradient accumulation of 4 (effective batch size 64), linear LR schedule with 1 000 warm‑up steps, and native AMP mixed‑precision. Training ran on 2 GPUs (multi‑GPU distributed type) for up to 100 epochs in the first stage and 60 epochs in the second.

The model remains fully fine‑tunable: users can continue training on domain‑specific Hebrew data, adjust the CTC decoder, or integrate a language model for beam search decoding.

Related Papers

The model builds on two seminal works:

• wav2vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations – introduces the self‑supervised encoder that powers the base model.
• XLS‑R: Self‑Supervised Learning for Cross‑Lingual Speech Representation – presents the multilingual pre‑training strategy behind the xls‑r‑300m checkpoint.

Both papers describe the transformer‑based architecture, contrastive pre‑training objectives, and the scaling laws that enable strong performance on low‑resource languages like Hebrew.

Licensing Information

The model card lists the license as unknown. In such cases the safest approach is to treat the model as all‑rights‑reserved until the author clarifies the terms. This means:

• Commercial use is not guaranteed. You should contact imvladikon or review the repository’s Hugging Face page for any hidden license files.
• Redistribution or inclusion in proprietary products may be restricted.
• Attribution is advisable: cite the model name, author, and the Hugging Face URL.
• Check the base model’s license (Facebook’s wav2vec2‑xls‑r‑300m is under MIT) – but note that fine‑tuning can introduce additional constraints.