wav2vec2-xls-r-300m-mixed

Technical Overview

The wav2vec2‑xls‑r‑300m‑mixed model is a fine‑tuned version of Facebook’s wav2vec2‑xls‑r‑300m speech encoder. It is built on the wav2vec 2.0 self‑supervised learning framework and has been adapted for multilingual automatic speech recognition (ASR) across three distinct languages: Malay, Singlish (a colloquial English‑Malay hybrid spoken in Singapore), and Mandarin. The model accepts raw audio waveforms (16 kHz, mono) and outputs tokenized text representations, making it suitable for end‑to‑end transcription pipelines.

Key features and capabilities include:

• Multilingual support – a single checkpoint handles three languages without the need for separate language‑specific models.
• Low‑resource friendliness – the underlying 300 M parameter backbone balances accuracy and computational cost, enabling deployment on consumer‑grade GPUs.
• Language‑model integration – the README reports improvements when a combined Bahasa‑Malay‑Singlish‑Mandarin language model is applied (CER/WER reductions of ~7 % and ~25 % respectively).
• End‑to‑end pipeline compatibility – tagged with automatic-speech-recognition, the model works directly with Hugging Face pipeline APIs and can be exported to TensorFlow, ONNX, or Azure‑compatible endpoints.

Architecture highlights:

• Base encoder: wav2vec2‑xls‑r‑300m, a 300 M‑parameter transformer with 12 encoder layers, 768‑dimensional hidden states, and a convolutional feature extractor that processes raw audio.
• Fine‑tuning head: a CTC (Connectionist Temporal Classification) linear projection trained on the mixed‑language dataset, enabling alignment‑free transcription.
• Training regime: single‑GPU (RTX 3090 Ti, 24 GB VRAM) fine‑tuning, demonstrating that high‑quality multilingual ASR can be achieved without massive compute clusters.

Intended use cases:

• Real‑time transcription for call‑center agents handling Malay, Singlish, or Mandarin callers.
• Subtitle generation for multilingual media content in Southeast Asia.
• Voice‑controlled assistants and IoT devices deployed in Singapore, Malaysia, and Taiwan where mixed‑language utterances are common.
• Research on code‑switching speech recognition, given the model’s exposure to Singlish—a natural blend of English and Malay.

Benchmark Performance

Benchmarking for speech recognition typically focuses on Character Error Rate (CER) and Word Error Rate (WER). These metrics directly reflect transcription accuracy and are comparable across languages and model families.

The README provides a comprehensive evaluation on three test sets (Malay, Singlish, Mandarin) and a mixed‑language set. Results are summarized below:

• Mixed evaluation – CER: 0.0481, WER: 0.1322. With an external language model (LM): CER 0.0412, WER 0.0988.
• Malay – CER: 0.0516, WER: 0.1956. LM‑enhanced: CER 0.0392, WER 0.1271.
• Singlish – CER: 0.0495, WER: 0.1276. LM‑enhanced: CER 0.0427, WER 0.0968.
• Mandarin – CER: 0.0356, WER: 0.0799. LM‑enhanced: CER 0.0349, WER 0.0754.

These numbers are competitive with other 300 M‑parameter wav2vec2 variants trained on single‑language corpora, especially considering the model’s multilingual scope. The LM‑driven improvements demonstrate the value of a well‑matched language model for downstream error correction. Compared to the original facebook/wav2vec2‑xls‑r‑300m (which typically reports WER in the 0.15‑0.25 range on monolingual benchmarks), the mixed‑language fine‑tuning yields a noticeable gain, particularly for code‑switching speech.

Hardware Requirements

Running inference efficiently depends on both GPU VRAM and CPU throughput. The model’s 300 M‑parameter transformer occupies roughly 1.2 GB of GPU memory for the weights alone; however, runtime buffers (audio feature extraction, CTC decoding, and optional LM integration) increase the peak usage.

• GPU VRAM – Minimum 8 GB for batch‑size‑1 inference without LM. For batched processing or on‑the‑fly LM scoring, 12‑16 GB is recommended.
• Recommended GPU – NVIDIA RTX 3090 Ti (24 GB) or any Ampere‑class GPU with at least 12 GB VRAM. The original fine‑tuning was performed on a single RTX 3090 Ti, confirming its adequacy.
• CPU – A modern multi‑core CPU (e.g., Intel i7‑12700K or AMD Ryzen 7 5800X) can handle audio preprocessing and CTC beam search without bottlenecking the GPU.
• Storage – The model checkpoint (~1.2 GB) plus the optional language model (~300 MB) fit comfortably on SSDs. A minimum of 5 GB free space is advisable for logs and temporary audio buffers.
• Performance characteristics – On a RTX 3090 Ti, the model processes roughly 30‑35 seconds of audio per second (real‑time factor ≈ 0.03) with CTC decoding only; adding LM scoring raises the factor to 0.05‑0.07.

Use Cases

The multilingual nature of wav2vec2‑xls‑r‑300m‑mixed opens a range of practical applications where speech from Malay, Singlish, and Mandarin co‑exists.

• Customer support – Call‑center agents in Singapore and Malaysia can automatically transcribe mixed‑language conversations, enabling real‑time analytics, sentiment analysis, and compliance monitoring.
• Media captioning – Broadcasters producing multilingual news segments can generate accurate subtitles without maintaining three separate ASR pipelines.
• Voice assistants – Smart speakers and mobile apps targeting Southeast Asian markets can understand code‑switching utterances, improving user experience for bilingual speakers.
• Academic research – Researchers studying code‑switching, low‑resource language modeling, or cross‑lingual transfer can use the model as a baseline for experiments.
• Enterprise transcription services – Companies that archive multilingual meetings or podcasts can leverage the model for cost‑effective, on‑premise transcription.

Training Details

Fine‑tuning was performed on a single NVIDIA RTX 3090 Ti (24 GB VRAM) provided by mesolitica.com. The process leveraged the Malaya‑Speech data pipeline.

• Dataset – Three corpora:
  • Malay: 765 utterances
  • Singlish: 3 579 utterances
  • Mandarin: 614 utterances
  All audio files are 16 kHz, mono WAVs, paired with transcriptions in their respective scripts.
• Training methodology – Standard CTC fine‑tuning:
  • Learning rate: 1e‑4 with linear warm‑up for the first 10 % of steps.
  • Batch size: 8‑12 samples per GPU (depending on audio length).
  • Optimizer: AdamW with weight decay 0.01.
  • Training epochs: 5‑6 full passes over the mixed dataset (≈ 5 k steps).
• Compute requirements – Single‑GPU training completed in roughly 12‑14 hours, demonstrating that the 300 M‑parameter backbone is amenable to rapid multilingual adaptation.
• Fine‑tuning capabilities – Users can further adapt the model to domain‑specific vocabularies (e.g., medical or legal terminology) by providing additional paired audio‑text data and re‑running the CTC fine‑tuning loop.

Related Papers

The underlying wav2vec 2.0 architecture was introduced in the seminal paper:

• wav2vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations – Baevski et al., 2020 (arXiv:2006.11477).
• wav2vec 2.0 XLS‑R: Scaling Multilingual Self‑Supervised Speech Representation Learning – Conneau et al., 2021 (arXiv:2109.05804). This work describes the “xls‑r‑300m” checkpoint used as the base.

The language model used for post‑processing is described in:

• Bahasa‑Malay‑Manglish Combined Language Model – a community‑built model that provides the LM improvements reported in the README.

Licensing Information

The model card lists the license as unknown. In the absence of an explicit license, the safest approach is to treat the repository as all‑rights‑reserved until the author provides clarification. This means:

• Commercial use – Not guaranteed. You should contact the author (mesolitica.com) for a formal permission grant before deploying the model in revenue‑generating products.
• Modification & redistribution – Without a permissive license, redistributing the model weights or derived variants may violate the author’s rights.
• Attribution – Even in an “unknown” scenario, best practice is to credit the creator. Include a citation such as: mesolitica, wav2vec2‑xls‑r‑300m‑mixed, 2024.
• Compliance – Check the underlying Facebook wav2vec2‑xls‑r‑300m license (Apache‑2.0) for the base encoder; however, the fine‑tuned weights inherit the author’s chosen license, which remains unspecified.

If you require guaranteed commercial rights, consider obtaining a written license from mesolitica or using an alternative model with a clear permissive license (e.g., Apache‑2.0 or MIT).