wav2vec2-xls-r-300m-bengali

Technical Overview

Model ID: arijitx/wav2vec2-xls-r-300m-bengali
Model Name: wav2vec2‑xls‑r‑300m‑bengali
Author: arijitx
Pipeline Tag: automatic‑speech‑recognition

This model is a Bengali‑language Automatic Speech Recognition (ASR) system built on top of Facebook’s wav2vec2‑xls‑r‑300m self‑supervised speech encoder. It has been fine‑tuned on the OpenSLR SLR53 Bengali corpus (≈ 10 k utterances) and further enhanced with a 5‑gram language model trained on 30 M sentences from the AI4Bharat IndicCorp dataset. The result is a compact, 300‑million‑parameter model that can transcribe spoken Bengali with high accuracy while remaining lightweight enough for on‑device or low‑latency server inference.

Key Features & Capabilities

• Supports the bn (Bengali) language out‑of‑the‑box.
• End‑to‑end speech‑to‑text conversion using the wav2vec2‑xls‑r architecture.
• Two operating modes: raw acoustic decoding (WER ≈ 21.7 %) and decoding with a 5‑gram LM (WER ≈ 15.3 %).
• Works with the Hugging Face transformers and datasets libraries, making integration into Python pipelines straightforward.
• Designed for robustness: training incorporated aggressive time‑masking (75 % probability) and feature‑masking (25 % probability) to improve performance on noisy or reverberant audio.

Architecture Highlights

• Base encoder: wav2vec2‑xls‑r‑300m (300 M parameters, convolutional feature extractor + Transformer encoder).
• Feature encoder frozen during fine‑tuning to retain the self‑supervised representations.
• Projection dropout set to 0 % (no additional noise) while activation dropout is 10 %.
• Masking strategy: mask_time_prob=0.75 with length 10, mask_feature_prob=0.25 with length 64, encouraging the model to learn context‑aware representations.
• Training stopped after 180 k steps, with checkpointing every 2 k steps and evaluation every 3 k steps.

Intended Use Cases

• Real‑time transcription of Bengali broadcast news, podcasts, and educational videos.
• Voice‑controlled applications for Bengali‑speaking users (e.g., virtual assistants, smart home devices).
• Subtitle generation for Bengali media content.
• Low‑resource deployment on edge devices where a 300 M‑parameter model fits comfortably in GPU memory.

Benchmark Performance

The most relevant benchmarks for an ASR model are Word Error Rate (WER) and Character Error Rate (CER). Both metrics are reported on the OpenSLR SLR53 test split (10 935 utterances, 5 % of the total corpus). The model achieves:

• Without language model: WER = 0.2173 (21.7 %), CER = 0.0473 (4.73 %).
• With a 5‑gram LM (trained on 30 M IndicCorp sentences): WER = 0.1532 (15.3 %), CER = 0.0341 (3.41 %).

These numbers are competitive for a 300 M‑parameter Bengali ASR system, especially considering the limited size of the OpenSLR dataset. The LM‑augmented results demonstrate that even a modest n‑gram model can substantially close the gap to human‑level transcription, highlighting the model’s suitability for production‑grade applications where post‑processing with a language model is feasible.

Hardware Requirements

The model’s size (≈ 300 M parameters) translates to roughly 1.2 GB of VRAM for the encoder weights alone when using 32‑bit floating point precision. In practice, inference can be performed comfortably on a single modern GPU with at least 4 GB of VRAM, such as an NVIDIA GTX 1650 or RTX 2060. For batch inference or higher throughput, a 6‑8 GB GPU (e.g., RTX 3060) is recommended.

• GPU: Minimum 4 GB VRAM; 6‑8 GB for batch sizes > 1.
• CPU: Any recent multi‑core CPU; inference speed improves with AVX‑512 support.
• Storage: Model checkpoint (~ 1.5 GB) plus LM files (~ 200 MB) – total < 2 GB.
• Performance: On a RTX 3060, single‑utterance latency is ~ 30 ms for a 5‑second audio clip (batch size = 1, fp16).

Use Cases

The model is purpose‑built for Bengali speech transcription, making it a strong candidate for a variety of real‑world applications:

• Media & Entertainment: Automatic subtitle generation for Bengali movies, TV shows, and YouTube videos.
• Education: Transcribing lecture recordings and creating searchable text archives for online courses.
• Customer Service: Voice‑to‑text logging for call‑center interactions in Bengali.
• Smart Devices: Voice command recognition for home assistants, automotive infotainment, and IoT devices targeting Bengali‑speaking markets.
• Research & Accessibility: Building corpora for linguistic research and providing speech‑to‑text services for the hearing‑impaired.

Training Details

Fine‑tuning was performed on the OpenSLR SLR53 Bengali dataset (≈ 10 k utterances). The first 95 % of the data were used for training and the final 5 % (10 935 examples) for evaluation. Training stopped after 180 k steps with a batch size of 32, a learning rate of 7.5 × 10⁻⁵, and a warm‑up of 2 000 steps. The feature encoder was frozen to preserve the self‑supervised speech representations, while the Transformer layers were updated with a 0.1 activation dropout and aggressive time‑masking (75 % probability). The training script was adapted from the Hugging Face robust‑speech‑event example.

• Datasets: OpenSLR SLR53 (audio + transcripts), AI4Bharat IndicCorp (LM training).
• Compute: Trained on a single GPU (e.g., NVIDIA V100) for ~ 180 k steps; total GPU time ≈ 24 hours.
• Fine‑tuning capability: Users can further fine‑tune on domain‑specific Bengali data by unfreezing the encoder or adjusting the LM.

Related Papers

The underlying architecture originates from the wav2vec 2.0 family, notably the wav2vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations (Baevski et al., 2020). The XLS‑R extension, which scales wav2vec 2.0 to 100+ languages, is described in XLS‑R: Large‑Scale Multilingual Self‑Supervised Speech Representation Learning (Babu et al., 2021). The fine‑tuning methodology follows the robust‑speech‑event framework from Hugging Face’s research examples, which in turn builds on the Robust Speech Event Detection (Kharitonov et al., 2021) paper.

Licensing Information

The repository’s license field is listed as “unknown”, but the model is tagged with license:apache‑2.0. Apache 2.0 is a permissive open‑source license that permits commercial use, modification, and distribution, provided that a copy of the license and a notice of any changes are included. Because the explicit license file is missing, users should treat the model as Apache 2.0‑compatible but verify the licensing status with the author before deploying in commercial products.

• Commercial use: Allowed under Apache 2.0, assuming the license is confirmed.
• Restrictions: No trademark use without permission; must retain the original copyright notice.
• Attribution: Cite the model card and the original wav2vec2‑xls‑r‑300m paper.
• Patents: Apache 2.0 includes an explicit patent‑grant, protecting downstream users.