wav2vec2-large-xlsr-marathi

Technical Overview

Model ID: sumedh/wav2vec2-large-xlsr-marathi
Model Name: wav2vec2-large-xlsr-marathi

This model is a Marathi‑specific automatic speech recognition (ASR) system built on top of Facebook’s wav2vec2‑large‑xlsr‑53 backbone. It transforms raw audio waveforms sampled at 16 kHz into Unicode text in the Devanagari script, delivering a word‑error‑rate (WER) of 12.7 % on the OpenSLR‑64 test split. The model is packaged as a Wav2Vec2ForCTC checkpoint together with a Wav2Vec2Processor that handles feature extraction, tokenization, and decoding.

Key Features & Capabilities

• Marathi‑focused fine‑tuning – Trained on the OpenSLR‑64 dataset (female‑only voices) but demonstrates solid performance on male speakers as well.
• End‑to‑end CTC decoding – No external language model is required for inference, simplifying deployment.
• High‑capacity encoder – 24‑layer Transformer with 1.5 B parameters, inheriting the robustness of the XLSR‑53 multilingual pre‑training.
• Audio preprocessing built‑in – Includes a 48 kHz → 16 kHz resampling step for compatibility with the original training data.
• Ready‑to‑use Hugging Face pipeline – Tagged with automatic-speech-recognition for one‑line inference via pipeline("automatic-speech-recognition").

Architecture Highlights

The underlying architecture follows the wav2vec 2.0 paradigm: a convolutional feature encoder extracts 16 kHz raw audio into latent speech representations, which are then fed into a deep Transformer encoder (24 layers, 1024 hidden size, 16 attention heads). The model is pre‑trained on 53 languages (XLSR) and subsequently fine‑tuned on Marathi using Connectionist Temporal Classification (CTC) loss. This two‑stage training yields a model that is both language‑agnostic in its low‑level acoustic modeling and highly specialized in its final transcription layer.

Intended Use Cases

• Voice assistants and chatbots that need to understand Marathi commands.
• Transcription services for podcasts, interviews, or educational content in Marathi.
• Assistive technologies for the hearing‑impaired community, providing real‑time captions.
• Data‑driven research on Marathi speech corpora, including dialect studies and phonetic analysis.

Benchmark Performance

For ASR models, the primary benchmark is the Word Error Rate (WER), which quantifies the edit distance between predicted and reference transcripts. The sumedh/wav2vec2-large-xlsr-marathi model achieves a 12.7 % WER on a 10 % held‑out test split of the OpenSLR‑64 dataset. This metric is derived from a reproducible evaluation script that normalizes punctuation, lower‑cases text, and resamples audio to 16 kHz.

Why this matters:

• WER directly reflects user‑perceived transcription quality; lower values mean fewer misrecognitions.
• OpenSLR‑64 is a publicly available, high‑quality Marathi speech corpus, making the benchmark comparable across research.
• The model’s performance rivals other multilingual XLSR‑based ASR systems while offering a language‑specific fine‑tune that reduces error rates by several points.

Compared to the base facebook/wav2vec2-large-xlsr-53 model (which typically reports >20 % WER on low‑resource languages), the Marathi‑fine‑tuned version demonstrates a substantial accuracy boost, confirming the value of targeted domain adaptation.

Hardware Requirements

• VRAM for inference: Approximately 4 GB is sufficient for batch size = 1 on a 16 kHz mono waveform. Larger batches (e.g., 8‑10 seconds) benefit from 6‑8 GB.
• Recommended GPU: NVIDIA RTX 3060 (12 GB) or higher; any GPU supporting CUDA 11+ and PyTorch 2.0 will run the model efficiently.
• CPU considerations: A modern multi‑core CPU (e.g., Intel i7‑10700K or AMD Ryzen 7 3700X) can handle real‑time streaming when paired with a GPU; pure CPU inference is possible but slower (≈150 ms per second of audio on a 12‑core machine).
• Storage: Model checkpoint (~1.2 GB) plus the OpenSLR‑64 dataset (~2 GB) for fine‑tuning. Disk I/O should be SSD for optimal loading times.
• Performance characteristics: On a RTX 3060, the model processes ~0.5 seconds of audio per millisecond of wall‑clock time (≈2× real‑time). Latency scales linearly with audio length.

Use Cases

Primary applications revolve around any scenario that requires converting spoken Marathi into text:

• Voice‑controlled smart devices – Enable Marathi language commands for home automation or IoT gadgets.
• Media transcription – Automatically generate subtitles for Marathi videos, news broadcasts, and podcasts.
• Customer support automation – Real‑time speech‑to‑text for call‑center agents, feeding downstream sentiment analysis pipelines.
• Educational tools – Provide live captions for online lectures, language learning apps, and accessibility tools for the hearing impaired.

Industries that can benefit include:

• Telecommunications – Voice‑bot integration for regional language support.
• Media & Entertainment – Captioning and content indexing.
• Healthcare – Documentation of patient interviews in Marathi.
• Government & Public Services – Speech‑enabled portals for citizen services.

Integration is straightforward via Hugging Face’s pipeline API, TorchScript export, or ONNX conversion for edge deployment.

Training Details

The fine‑tuning process leveraged the OpenSLR‑64 Marathi corpus, which contains female‑only voice recordings. The dataset was split 90 % for training and 10 % for testing, with a fixed random seed (2020) to ensure reproducible WER scores.

Key training configuration:

• Compute platform: Google Colab Pro with a Tesla P100 16 GB GPU.
• Batch size: 8‑12 samples per GPU (adjusted for memory constraints).
• Learning rate schedule: Linear warm‑up followed by cosine decay, typical for wav2vec 2.0 fine‑tuning.
• Loss: CTC loss computed on tokenized Devanagari characters.
• Framework: PyTorch + 🤗 Transformers, with Wav2Vec2Processor handling feature extraction.

Fine‑tuning logs and experiment tracking are available via Weights & Biases (run summary). The model can be further adapted to other Marathi datasets or extended with a language model for improved punctuation and capitalization.

Related Papers

The model builds on the seminal wav2vec 2.0 architecture introduced by Baevski et al. (2020). Key references include:

• wav2vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations – The foundational self‑supervised pre‑training method.
• XLSR (Cross‑Lingual Speech Representation) paper – Describes multilingual pre‑training on 53 languages, the exact base model used here.

These works provide the theoretical underpinnings for the multilingual acoustic encoder and the CTC fine‑tuning approach applied to Marathi.

Licensing Information

The model is released under the Apache‑2.0 license, as indicated in the README. Apache‑2.0 is a permissive open‑source license that grants:

• Freedom to use, modify, and distribute the model for both commercial and non‑commercial purposes.
• No requirement to disclose source code when integrating the model into proprietary products.
• Obligation to retain the original copyright notice and provide a copy of the license.
• Patent‑grant provisions that protect downstream users from patent litigation related to the contributed code.

Because the underlying dataset (OpenSLR‑64) is also under a permissive license, you can safely embed this ASR system in commercial applications such as call‑center analytics, mobile voice assistants, or SaaS transcription platforms. The only mandatory step is proper attribution to the original authors (Sumedh Khodke) and to Facebook’s wav2vec 2.0 model.