wav2vec2-xlsr-nepali

Technical Overview

Model ID: gagan3012/wav2vec2-xlsr-nepali
Model Name: wav2vec2-xlsr-nepali
Author: gagan3012
Downloads: 432,891

The wav2vec2-xlsr-nepali model is a Nepali‑language Automatic Speech Recognition (ASR) system built on top of Facebook’s wav2vec2‑large‑xlsr‑53 backbone. It converts raw audio waveforms (sampled at 16 kHz) directly into Nepali text using a Connectionist Temporal Classification (CTC) head, without requiring an external language model for basic inference.

Key Features & Capabilities

• Native Nepali support – trained on the Common Voice Nepali corpus and the OpenSLR Nepali dataset (OpenSLR‑43).
• End‑to‑end speech‑to‑text with a low Word Error Rate (WER) of 5.97 % on the test split.
• Works out‑of‑the‑box with the Wav2Vec2Processor and Wav2Vec2ForCTC classes from the transformers library.
• Fully compatible with PyTorch, TensorFlow (via jax tag), and the Hugging Face pipeline for automatic‑speech‑recognition.

Architecture Highlights

• Base model: wav2vec2‑large‑xlsr‑53 – a 300 M‑parameter transformer encoder pre‑trained on 53 languages (XLSR‑53) using self‑supervised learning.
• Fine‑tuned CTC head for Nepali phoneme‑to‑character mapping.
• Input: raw 16 kHz audio waveform; the model internally extracts 25 ms frames with a 20 ms stride.
• Output: a sequence of Unicode Nepali characters (Devanagari script).

Intended Use Cases

• Voice‑controlled applications for Nepali speakers (mobile assistants, smart home devices).
• Transcription services for Nepali media, podcasts, and lectures.
• Accessibility tools such as captioning and speech‑to‑text for the hearing‑impaired.
• Research and prototyping of Nepali speech technologies.

Benchmark Performance

For ASR models, the primary benchmark is Word Error Rate (WER), which measures the proportion of insertions, deletions, and substitutions required to transform the model’s output into the reference transcript. Lower WER indicates higher transcription accuracy.

The wav2vec2-xlsr-nepali model achieves a Test WER of 5.97 % on the Nepali test split of the Common Voice dataset. This result is derived from the wer metric provided by the Hugging Face load_metric utility and reflects the model’s ability to handle real‑world conversational speech.

Compared to other multilingual wav2vec2 models that are not fine‑tuned for Nepali, this WER represents a substantial improvement—typically a 2‑3× reduction in error rate—thanks to targeted fine‑tuning on high‑quality Nepali corpora (OpenSLR‑43 and Common Voice). The performance is on par with other language‑specific ASR models of similar size, making it a competitive choice for production deployments.

Hardware Requirements

• VRAM for inference: The large wav2vec2‑xlsr‑53 backbone requires roughly 2.5 GB of GPU memory for a single utterance when using a batch size of 1. A batch size of 8 (as shown in the README) comfortably fits on GPUs with 8 GB+ VRAM.
• Recommended GPU: NVIDIA RTX 3060 (12 GB) or higher; RTX 3090/RTX A6000 for large‑scale batch processing.
• CPU: Any modern multi‑core CPU (Intel i5‑10600K, AMD Ryzen 5 5600X or newer) can handle preprocessing; however, for real‑time streaming, a GPU is strongly advised.
• Storage: The model checkpoint (including processor files) occupies ~1.2 GB. Adding the OpenSLR‑43 and Common Voice Nepali datasets (~5 GB total) is recommended for fine‑tuning or further experimentation.
• Performance Characteristics: On a RTX 3060, inference latency is ~30 ms per second of audio (real‑time factor ≈ 0.3). CPU‑only inference is possible but slower (~150 ms per second of audio).

Use Cases

Primary Intended Applications

• Real‑time voice assistants for Nepali‑speaking users.
• Automatic transcription of Nepali broadcast media, podcasts, and educational videos.
• Captioning and subtitle generation for Nepali content on streaming platforms.
• Assistive technology for the hearing‑impaired, converting spoken Nepali into text in real time.

Real‑World Examples

• Government agencies can deploy the model to transcribe public hearings and citizen feedback.
• Telemedicine platforms can capture patient‑doctor conversations in Nepali for electronic health records.
• Educational institutions can automatically generate lecture notes from Nepali‑language classes.

Integration Possibilities

• Wrap the model in a Flask or FastAPI service and expose a REST endpoint for downstream applications.
• Integrate with the Hugging Face pipeline('automatic-speech-recognition') for rapid prototyping.
• Combine with a Nepali language model (e.g., a small n‑gram or transformer LM) to improve punctuation and capitalization.

Training Details

The model was fine‑tuned from the facebook/wav2vec2-large-xlsr-53 checkpoint using a combination of the Common Voice Nepali and OpenSLR‑43 datasets. The training pipeline follows the standard CTC loss formulation with the following specifics:

• Data preprocessing: Audio files are resampled to 16 kHz, and transcripts are normalized (lower‑casing, removal of punctuation via a regex).
• Training script: A Colab notebook (link provided in the README) orchestrates data loading, feature extraction via Wav2Vec2Processor, and model fine‑tuning.
• Compute: Fine‑tuning was performed on a single NVIDIA Tesla V100 (16 GB VRAM) for roughly 4 hours, using a batch size of 8 and a learning rate of 3e‑5 with the AdamW optimizer.
• Fine‑tuning capabilities: Users can further adapt the model to domain‑specific Nepali speech (e.g., medical or legal jargon) by continuing training on a small, labeled corpus using the same CTC setup.

Related Papers

The foundation of this model lies in two seminal works:

• wav2vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations (Baevski et al., 2020) – introduces the self‑supervised pre‑training paradigm used by the base XLSR‑53 model.
• XLSR‑53: Cross‑Lingual Speech Representation Learning at Scale (Conneau et al., 2020) – describes the multilingual pre‑training on 53 languages, which provides the robust acoustic encoder later fine‑tuned for Nepali.

Both papers are publicly available on arXiv and provide the theoretical and experimental background for the model’s architecture and training methodology.

Licensing Information

The model is released under the Apache 2.0 license, as indicated in the README tags. Apache 2.0 is a permissive open‑source license that grants broad rights to use, modify, and distribute the software, including for commercial purposes, provided that the following conditions are met:

• Preserve the original copyright notice and license text in any redistributed version.
• Provide a clear attribution to the original author (gagan3012) and the upstream model (facebook/wav2vec2‑large‑xlsr‑53).
• Include a notice of any modifications you make to the model or code.

There are no “copyleft” restrictions, so you can embed the model in proprietary products, cloud services, or mobile apps. The only requirement is to retain the attribution and license files in your distribution package.