wav2vec2-base-vi-vlsp2020

Technical Overview

The nguyenvulebinh/wav2vec2-base-vi-vlsp2020 model is a Hugging Face model card that implements the wav2vec 2.0 architecture for automatic speech recognition (ASR) in Vietnamese (vi). It converts raw 16 kHz audio waveforms directly into text without any intermediate phoneme or feature extraction steps, making it ideal for end‑to‑end speech‑to‑text pipelines.

Key features and capabilities include:

• Pre‑trained on 13 000 hours of unlabeled Vietnamese YouTube audio, providing a robust acoustic representation of the language.
• Fine‑tuned on 250 hours of manually transcribed speech from the VLSP ASR 2020 dataset, achieving state‑of‑the‑art word‑error‑rate (WER) for a base‑size model.
• Supports Wav2Vec2ProcessorWithLM for decoding with an optional 5‑gram language model, dramatically improving transcription quality.
• Fully compatible with the automatic‑speech‑recognition pipeline tag in the 🤗 Transformers library.
• Open‑source code for model handling (see model_handling.py) and a ready‑to‑run Colab notebook.

Architecture highlights:

• Base configuration: 12 Transformer encoder layers, 768 hidden dimensions, 12 attention heads, and 300 M trainable parameters.
• Feature extractor operates on raw waveform, producing 512‑dimensional latent representations that are fed to the CTC decoder.
• CTC loss is used during fine‑tuning, enabling alignment‑free training on the VLSP corpus.
• Optional integration with KenLM and pyctcdecode for beam search with a 5‑gram language model.

Intended use cases revolve around any application that requires Vietnamese speech transcription: voice assistants, call‑center analytics, media captioning, subtitle generation, and low‑resource speech research. The model’s base size strikes a balance between accuracy and compute efficiency, making it suitable for on‑device inference on modern GPUs or even high‑end CPUs.

Benchmark Performance

For Vietnamese ASR, the most informative benchmark is Word Error Rate (WER) on a held‑out test set. The model is evaluated on the VLSP T1 test split, a standard benchmark for Vietnamese speech recognition.

The 8.66 % WER without a language model already outperforms many earlier Vietnamese ASR systems that were built on traditional HMM‑DNN pipelines. Adding a 5‑gram LM drops the error to 6.53 %, a 25 % relative improvement, demonstrating the value of linguistic context for Vietnamese, which has many homophones.

Compared to the large‑size counterpart (6.90 % / 5.32 %), the base model offers a modest increase in error while requiring roughly half the memory and compute. This makes it the preferred choice when deployment constraints dominate.

Hardware Requirements

Inference speed and memory consumption depend on the underlying hardware. The following guidelines are based on typical usage with the 🤗 Transformers library (v4.20.0) and a batch size of one 16‑second audio clip.

• VRAM for inference: ~3 GB (FP32) or ~1.5 GB (FP16) for the base model. A GPU with at least 4 GB of memory is recommended to leave headroom for the language model and batch processing.
• Recommended GPU: NVIDIA RTX 2070/3080, AMD Radeon RX 6700 XT, or any GPU with 8 GB+ VRAM for comfortable beam‑search decoding.
• CPU requirements: A modern multi‑core CPU (e.g., Intel i7‑9700K, AMD Ryzen 7 3700X) can run the model in real time for short utterances when using FP16 or ONNX‑runtime optimizations.
• Storage needs: The model checkpoint (weights + tokenizer + LM) occupies roughly 500 MB. Including the optional KenLM 5‑gram model adds another ~150 MB.
• Performance characteristics: On an RTX 3080, a 16‑second audio file is transcribed in ~0.25 seconds (≈64× real‑time) without LM and ~0.45 seconds with a 5‑gram LM (beam width = 100).

Use Cases

The wav2vec2‑base‑vi‑vlsp2020 model is engineered for real‑world Vietnamese speech‑to‑text applications. Below are some concrete scenarios:

• Voice‑controlled assistants: Integrate the model into smart speakers or mobile apps to enable Vietnamese voice commands.
• Call‑center analytics: Transcribe inbound and outbound calls for sentiment analysis, keyword spotting, and quality assurance.
• Media captioning: Automatically generate subtitles for Vietnamese podcasts, YouTube videos, and live broadcasts.
• Educational tools: Provide instant feedback on pronunciation for language‑learning platforms.
• Research & development: Serve as a baseline for low‑resource speech research, domain adaptation, or multilingual ASR experiments.

Because the model is compatible with the 🤗 Transformers automatic‑speech‑recognition pipeline, integration with Python, JavaScript (via ONNX), or even edge‑device runtimes (TensorFlow Lite, TorchScript) is straightforward.

Training Details

Training consisted of two stages:

• Self‑supervised pre‑training: 13 000 hours of unlabelled Vietnamese YouTube audio (16 kHz) were used to train a wav2vec 2.0 base encoder. The objective was contrastive loss over masked latent speech, following the methodology of Baevski et al.
• Supervised fine‑tuning: The pre‑trained encoder was fine‑tuned on the VLSP‑ASR‑2020 corpus (≈250 hours, 16 kHz) using Connectionist Temporal Classification (CTC) loss. Training was performed with the 🤗 Transformers library (v4.20.0) on a single NVIDIA V100 GPU for ~48 hours.

Key hyper‑parameters (as reported by the author’s GitHub repo):

• Learning rate: 3e‑5 (linear warm‑up for 10 k steps, then cosine decay)
• Batch size: 8 seconds of audio per GPU (effective batch ≈ 64 seconds)
• Optimizer: AdamW with β₁=0.9, β₂=0.98
• Gradient clipping at 1.0

The model can be further fine‑tuned on domain‑specific Vietnamese corpora (e.g., medical or legal speech) by loading the base checkpoint with Wav2Vec2ForCTC.from_pretrained and continuing CTC training. The provided model_handling.py script simplifies this process.

Related Papers

The core technology behind this model is described in the seminal wav2vec 2.0 paper (Baevski et al., 2020). The authors introduced a self‑supervised pre‑training objective that learns speech representations from raw audio, which is exactly the approach used for the 13 k‑hour Vietnamese YouTube pre‑training.

The VLSP ASR 2020 dataset, on which the model is fine‑tuned, is documented in the VLSP workshop proceedings (see VLSP 2020 paper – placeholder DOI). This dataset provides high‑quality transcriptions for a diverse set of Vietnamese speakers, making it a reliable benchmark for ASR research.

Licensing Information

The model’s parameters are released under the Creative Commons Attribution‑NonCommercial 4.0 International (CC BY‑NC 4.0) license. This license permits anyone to share and adapt the model for non‑commercial purposes, provided that proper attribution is given to the author (nguyenvulebinh).

• Commercial use: Not allowed under CC BY‑NC 4.0. If you intend to embed the model in a product that generates revenue, you must obtain a separate commercial license from the author.
• Attribution requirement: Cite the model name, author, and a link to the Hugging Face model card. A typical citation could be: “nguyenvulebinh/wav2vec2-base-vi-vlsp2020 – Vietnamese ASR, CC BY‑NC 4.0”.
• Derivative works: You may fine‑tune or modify the model for research or personal projects, but you must keep the same non‑commercial restriction on any distribution of the derived model.
• Other restrictions: The license does not cover the underlying datasets (YouTube audio and VLSP ASR 2020). Users should verify that their use complies with the dataset’s own licenses (YouTube’s terms of service and CC BY‑NC 4.0 for VLSP).