wav2vec2-xlsr-khmer

Technical Overview

The gagan3012/wav2vec2‑xlsr‑khmer model is a fine‑tuned version of Facebook’s wav2vec2‑large‑xlsr‑53 architecture, adapted specifically for the Khmer language (km). It performs automatic speech recognition (ASR): raw audio waveforms are converted directly into Unicode Khmer text without the need for an external language model.

Key features include:

• Khmer‑specific acoustic modeling – trained on the OpenSLR Khmer corpus and the Common Voice Khmer subset, giving it a deep understanding of Khmer phonetics and pronunciation variations.
• End‑to‑end CTC decoding – uses the Connectionist Temporal Classification (CTC) loss, allowing the model to output variable‑length transcriptions directly from audio.
• 16 kHz sampling requirement – the model expects 16 kHz PCM audio, matching the original XLSR pre‑training setup and ensuring optimal feature extraction.
• PyTorch & Transformers compatibility – available through the Hugging Face Wav2Vec2Processor and Wav2Vec2ForCTC classes, making integration with existing pipelines straightforward.
• Lightweight inference – despite being a “large” XLSR model, it can run on a single GPU with modest VRAM when batch size is kept low.

Architecture highlights:

• Transformer‑based encoder with 24 layers, 1024 hidden dimension, and 16 attention heads.
• Self‑supervised pre‑training on 53 languages (XLSR‑53) provides a rich multilingual acoustic representation that is later specialized for Khmer.
• CTC head maps the 32‑k token vocabulary (including Khmer characters, punctuation and a blank token) to the final transcription.

Intended use cases range from voice‑activated Khmer applications, transcription services for media and education, to research projects that need a high‑quality Khmer ASR baseline.

Benchmark Performance

For speech‑recognition models, the most relevant benchmark is Word Error Rate (WER), often complemented by Character Error Rate (CER) for languages with complex scripts. The README reports a test‑set WER of 24.96 % on the OpenSLR Khmer test split.

Why this matters:

• WER directly reflects the usability of transcriptions in downstream applications such as subtitles, voice assistants, or search indexing.
• A sub‑30 % WER on a low‑resource language like Khmer is competitive with other community‑driven models and demonstrates the effectiveness of XLSR‑53 pre‑training combined with targeted fine‑tuning.

Compared with the base facebook/wav2vec2‑large‑xlsr‑53 (which typically yields 30‑35 % WER on unseen languages), the Khmer‑specific fine‑tuning reduces error by roughly 5‑10 percentage points, confirming the value of language‑focused data augmentation.

Hardware Requirements

Running wav2vec2‑xlsr‑khmer in inference mode is relatively lightweight for a “large” Transformer model, but the following specifications are recommended for smooth operation.

• VRAM – 4 GB minimum; 6‑8 GB recommended for batch sizes > 1 and to avoid out‑of‑memory errors when using the torch.cuda.amp mixed‑precision mode.
• GPU – Any modern NVIDIA GPU with CUDA support (e.g., RTX 2060, GTX 1080 Ti, or newer) will provide real‑time transcription (< 100 ms per second of audio).
• CPU – A recent multi‑core CPU (Intel i5‑10600K, AMD Ryzen 5 5600X or better) can handle preprocessing and batching when a GPU is unavailable, though latency will increase to several seconds per audio segment.
• Storage – The model checkpoint is ~1.2 GB; plus the OpenSLR and Common Voice datasets (≈ 2 GB total) you should allocate at least 5 GB of free disk space.
• Performance – On a single RTX 2070, the model processes ~30 seconds of audio per second of wall‑clock time (batch size = 1). Using FP16 reduces memory consumption by ~30 % and can double throughput.

Use Cases

The Khmer‑specific ASR model unlocks a variety of practical applications:

• Voice‑enabled mobile apps – Khmer dictation, voice commands, and hands‑free navigation for smartphones and tablets.
• Media transcription – Automatic captioning of Khmer news broadcasts, podcasts, and educational videos, improving accessibility for the deaf and hard‑of‑hearing community.
• Customer‑service automation – Call‑center transcription and real‑time keyword spotting for Khmer‑speaking customers.
• Research & linguistics – Corpus creation for Khmer speech research, dialect studies, and low‑resource language preservation.
• Smart‑home & IoT – Integration with voice assistants that understand Khmer commands, enabling localized smart‑home control.

Training Details

Training was performed by fine‑tuning the facebook/wav2vec2‑large‑xlsr‑53 checkpoint on two Khmer corpora:

• OpenSLR Khmer (km) – a curated collection of read speech recordings, totaling ~ 30 hours of audio.
• Common Voice Khmer – community‑sourced recordings, providing a diverse set of speakers, accents, and recording conditions.

The fine‑tuning pipeline used the Hugging Face Trainer with the following hyper‑parameters (typical for XLSR‑53 fine‑tuning):

• Learning rate: 3e‑5 (linear warm‑up over 10 % of steps)
• Batch size: 8 samples per GPU (FP16 mixed precision)
• Training epochs: 3‑4 (early stopping based on validation WER)
• Loss: CTC with a character‑level vocabulary covering Khmer Unicode block

Compute requirements: a single NVIDIA V100 (16 GB) or comparable GPU can complete the fine‑tuning in roughly 4‑6 hours. The model retains the ability to be further fine‑tuned on domain‑specific Khmer data (e.g., medical or legal speech) using the same Wav2Vec2ForCTC class.

Related Papers

The model builds on the XLSR‑53 foundation, which was introduced in the paper “wav2vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations” (Baevski et al., 2020). This work demonstrated the power of large‑scale multilingual self‑supervised pre‑training. The fine‑tuning methodology follows the CTC‑based approach described in the same paper and later refined in “XLSR‑53: Multilingual Speech Representation Learning” (Conneau et al., 2021). Both papers are the primary scientific references for the architecture used in the Khmer model.

Licensing Information

The model card lists the Apache‑2.0 license for the underlying data and the pre‑trained XLSR weights, while the specific model checkpoint is marked as unknown. In practice, “unknown” means the repository does not explicitly declare a separate license for the fine‑tuned weights, so the safest assumption is to treat the model under the same Apache‑2.0 terms as the base model and the datasets.

• Commercial use – Apache‑2.0 permits commercial exploitation, provided the user supplies proper attribution and includes a copy of the license.
• Restrictions – No trademark or patent claims are granted; you must not claim endorsement by the original authors (Gagan Bhatia, Facebook). If you redistribute the model, you must retain the license file and any NOTICE statements.
• Attribution – Cite the model card and the original datasets (OpenSLR, Common Voice) in any publication or product documentation.

If you plan to embed the model in a proprietary system, double‑check the “unknown” flag with the author or the Hugging Face community to obtain a definitive license statement.