wav2vec2-indonesian-javanese-sundanese

Technical Overview

The wav2vec2-indonesian-javanese-sundanese model (ID: indonesian-nlp/wav2vec2-indonesian-javanese-sundanese) is a multilingual automatic‑speech‑recognition (ASR) system built for the three major languages of Indonesia: Bahasa Indonesia (id), Javanese (jv) and Sundanese (sun). It takes raw audio sampled at 16 kHz and outputs a sequence of characters (CTC decoding) without requiring an external language model. The model is a fine‑tuned version of Facebook’s wav2vec2‑large‑xlsr‑53, a self‑supervised transformer that was pre‑trained on 53 languages and then adapted to the target Indonesian languages using high‑quality speech corpora.

Key features and capabilities

• Supports three languages in a single checkpoint, enabling code‑switching and multilingual inference.
• Works out‑of‑the‑box with the Wav2Vec2Processor and Wav2Vec2ForCTC classes from 🤗 Transformers.
• Optimized for 16 kHz PCM audio (the model internally expects this sampling rate).
• Provides competitive word‑error‑rate (WER) and character‑error‑rate (CER) scores on Common Voice and Robust Speech Event datasets.
• Open‑source and compatible with Hugging Face Spaces for live demo testing.

Architecture highlights

• Backbone: 24‑layer transformer encoder (≈ 317 M parameters) with a 12 kHz convolutional front‑end that extracts latent speech representations.
• CTC head: Linear projection to 32 k character vocabulary (including language‑specific symbols).
• Fine‑tuned on a mixture of public Common Voice recordings and high‑quality TTS datasets (SLR41 for Javanese, SLR44 for Sundanese) to improve robustness across accents and recording conditions.

Intended use cases

• Voice‑enabled mobile and web applications targeting Indonesian users.
• Transcription of multilingual podcasts, news broadcasts, and educational content.
• Assistive technologies for speakers of Javanese and Sundanese.
• Research on low‑resource ASR and code‑switching speech recognition.

Benchmark Performance

Benchmarking for ASR models typically focuses on Word Error Rate (WER) and Character Error Rate (CER), as they directly reflect transcription quality. The wav2vec2-indonesian-javanese-sundanese model has been evaluated on several public datasets:

• Common Voice 6.1 (Indonesian) – WER = 4.056 % & CER = 1.472 %.
• Common Voice 7 (Indonesian) – WER = 4.492 % & CER = 1.577 %.
• Robust Speech Event – Dev – WER ≈ 48.94 % (noisy, far‑field recordings).
• Robust Speech Event – Test – WER ≈ 68.95 % (challenging acoustic conditions).

These numbers illustrate that the model excels on clean, crowd‑sourced speech (Common Voice) while still maintaining reasonable performance under adverse conditions. Compared to the base wav2vec2‑large‑xlsr‑53 model, the fine‑tuned version achieves a 2‑3 % absolute WER reduction on Indonesian data, confirming the value of language‑specific adaptation.

Hardware Requirements

Inference with a 317 M‑parameter wav2vec2 model is memory‑intensive but can be run on a wide range of hardware. Below are practical guidelines:

• GPU VRAM: Minimum 8 GB (e.g., NVIDIA RTX 2070) for batch size = 1; 12 GB+ (RTX 3080, A100) recommended for larger batches or real‑time streaming.
• GPU Architecture: Any CUDA‑compatible GPU with Compute Capability ≥ 6.0; Tensor cores accelerate the matrix multiplications in the transformer.
• CPU: Modern multi‑core CPUs (Intel i5‑10600K, AMD Ryzen 5 5600X) can handle preprocessing and inference when GPU is unavailable, though latency will increase to ~1–2 seconds per 5‑second audio clip.
• RAM: 8 GB system RAM is sufficient for loading the model and audio buffers.
• Storage: Model checkpoint (~1.2 GB) plus associated tokenizer files; SSD storage improves loading times.
• Performance: On a RTX 3080, the model processes ~30 ms of audio per millisecond of wall‑clock time (≈ 30× real‑time) for 16 kHz input.

Use Cases

The multilingual nature of this ASR system opens several practical applications:

• Voice assistants for Indonesian markets that need to understand Bahasa Indonesia, Javanese, and Sundanese interchangeably.
• Transcription services for media houses, NGOs, and government agencies that archive multilingual audio recordings.
• Educational platforms that provide subtitles or speech‑to‑text for e‑learning content in regional languages.
• Call‑center analytics to automatically log and analyze customer calls in multiple local languages.
• Research tools for linguists studying code‑switching, dialectal variation, and low‑resource language modeling.

Training Details

The model was fine‑tuned from the wav2vec2‑large‑xlsr‑53 checkpoint using a mixture of public and proprietary datasets:

• Datasets: Mozilla Common Voice 7.0 (Indonesian), OpenSLR SLR41 (Javanese TTS), OpenSLR SLR44 (Sundanese TTS), Magic Data, and TITML.
• Training regime: CTC loss with a learning rate of 5e‑5, batch size of 8–16 (depending on GPU memory), and early stopping based on validation WER.
• Compute: Approximately 48 hours on a single NVIDIA A100 (40 GB) GPU; the total FLOPs are comparable to other wav2vec2 fine‑tuning runs.
• Fine‑tuning capabilities: Users can further adapt the model to domain‑specific vocabularies (e.g., medical or legal) by continuing CTC training on a small labeled corpus.

All preprocessing steps (resampling to 16 kHz, text normalization) follow the scripts provided in the README, ensuring reproducibility across environments.

Related Papers

The model builds upon two seminal works:

• wav2vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations – Introduces the self‑supervised pre‑training paradigm used in the base model.
• XLSR‑53: Cross‑Lingual Speech Representation Learning at Scale – Describes the multilingual pre‑training of the wav2vec2‑large‑xlsr‑53 checkpoint that serves as the backbone.

Fine‑tuning on Indonesian, Javanese, and Sundanese data follows the methodology outlined in the original wav2vec 2.0 paper, adapting the CTC head to language‑specific vocabularies.

Licensing Information

The model card lists the Apache‑2.0 license, which is a permissive open‑source license. Under Apache‑2.0 you may:

• Use the model for commercial and non‑commercial purposes without paying royalties.
• Modify, redistribute, and embed the model in software products.
• Combine the model with other code under different licenses, provided you retain the original copyright notice and license text.

Key requirements:

• Include a copy of the Apache‑2.0 license in any distribution that contains the model.
• Provide proper attribution to the original creators (Indonesian‑NLP) and the upstream wav2vec2‑large‑xlsr‑53 model.
• State any modifications you make to the model or its weights.

Because the license is explicit, there are no hidden restrictions on commercial deployment, making the model suitable for products, services, and research alike.