wav2vec2-large-xlsr-53-japanese

Technical Overview

Model ID: jonatasgrosman/wav2vec2-large-xlsr-53-japanese
Model Name: wav2vec2‑large‑xlsr‑53‑japanese
Author: Jonatas Grosman
Pipeline Tag: automatic‑speech‑recognition (ASR)

This model is a Japanese‑specific fine‑tune of Facebook’s wav2vec2‑large‑xlsr‑53 self‑supervised speech encoder. It converts raw audio waveforms sampled at 16 kHz into a sequence of characters using a CTC (Connectionist Temporal Classification) head, delivering end‑to‑end speech‑to‑text without the need for an external language model.

• Key features
  • Pre‑trained on 53 k‑hours of multilingual audio (XLS‑R) → strong cross‑lingual acoustic representations.
  • Fine‑tuned on Japanese corpora (Common Voice ja, CSS10, JSUT) → vocab tailored to Japanese kana, kanji and punctuation.
  • Supports both torch and jax back‑ends; compatible with Hugging Face transformers and the Huggingsound library for one‑line inference.
  • Works out‑of‑the‑box with 16 kHz audio; no additional resampling required.
• Architecture highlights
  • Base encoder: Wav2Vec2Model with 24 transformer blocks, 1024 hidden size, 16 attention heads (≈ 300 M parameters).
  • CTC head: linear projection to a 32‑character Japanese vocabulary (including ▁ word‑boundary token).
  • Training objective: CTC loss on token‑level transcripts.
  • Fine‑tuning strategy: layer‑wise learning‑rate decay, gradient clipping, and mixed‑precision (FP16) for efficient GPU usage.
• Intended use cases
  • Real‑time transcription of Japanese podcasts, webinars, and meetings.
  • Closed‑caption generation for streaming platforms targeting Japanese audiences.
  • Voice‑controlled assistants and smart‑home devices speaking Japanese.
  • Data‑annotation pipelines for Japanese speech corpora (e.g., creating subtitles or training data for downstream NLP tasks).

Benchmark Performance

The model is evaluated on the Japanese test split of Common Voice ja. Two standard ASR metrics are reported:

• WER (Word Error Rate): 81.80 %
• CER (Character Error Rate): 20.16 %

WER measures how many words are incorrectly transcribed, while CER is more fine‑grained for Japanese where a single word may contain multiple characters. A CER around 20 % is competitive for a model trained without an external language model, especially given the diversity of the three training corpora (Common Voice, CSS10, JSUT). Compared to the vanilla wav2vec2‑large‑xlsr‑53 model (which reports > 30 % CER on Japanese), this fine‑tuned version shows a clear improvement, confirming the value of domain‑specific adaptation.

Hardware Requirements

• VRAM for inference: ~ 4 GB (FP16) – the model fits comfortably on most consumer‑grade GPUs (e.g., RTX 3060, RTX 2070).
• Recommended GPU: NVIDIA GeForce RTX 3060 Ti or higher, with at least 8 GB VRAM for batch processing of longer audio segments.
• CPU: Any modern x86_64 CPU; inference can be run on CPU‑only at ~ 0.5 × real‑time speed (≈ 2 seconds per second of audio).
• Storage: Model checkpoint ~ 1.2 GB (including tokenizer and config files). Additional space needed for audio datasets (≈ 10 GB for a small test set).
• Performance characteristics: On a single RTX 3060, the model processes ~ 30 seconds of audio per second (real‑time factor ≈ 0.03) when using the torchscript or onnx exported versions.

Use Cases

• Media & Entertainment: Automatic subtitle generation for Japanese TV shows, movies, and YouTube videos.
• Enterprise Collaboration: Real‑time transcription of Japanese conference calls and internal webinars.
• Assistive Technology: Speech‑to‑text for hearing‑impaired users or voice‑controlled applications on smartphones and smart‑home devices.
• Research & Data Collection: Rapid annotation of Japanese speech corpora for downstream NLP tasks such as sentiment analysis or machine translation.

Training Details

The model was fine‑tuned from the base facebook/wav2vec2-large-xlsr-53 checkpoint using the following pipeline:

• Datasets: Common Voice ja (train/validation splits), CSS10, JSUT – all resampled to 16 kHz.
• Pre‑processing: Audio normalized to ‑25 dBFS, transcripts upper‑cased, punctuation stripped for CTC alignment.
• Training script: wav2vec2‑sprint (PyTorch + Hugging Face 🤗 Accelerate).
• Compute: 2 × NVIDIA A100 40 GB GPUs, mixed‑precision (FP16), total wall‑time ≈ 12 hours.
• Hyper‑parameters: Learning rate 5e‑5 (warm‑up 10 % of steps), batch size 16, CTC loss, AdamW optimizer, gradient clipping at 1.0.
• Fine‑tuning capabilities: The checkpoint can be further fine‑tuned on domain‑specific Japanese speech (e.g., medical or legal vocabularies) by replacing the tokenizer’s vocab or adding a language model for post‑processing.

Related Papers

The foundation of this model is the XLS‑R (Cross‑Lingual Speech Representation) approach introduced in the paper:

• Alexei Baevski et al., “XLS‑R: Self‑Supervised Learning for Cross‑Lingual Speech Representation”, arXiv:2111.09296.

Fine‑tuning on Japanese leverages the Common Voice dataset (doi:10.57967/hf/3568) and the JSUT corpus (Shinnosuke Kamisawa et al., “JSUT: Japanese Speech Corpus of Pronunciation and Transcription”, Interspeech 2017). These resources provide the phonetic diversity needed for robust Japanese ASR.

Licensing Information

The model card lists the license as Apache‑2.0, which is a permissive open‑source license. This grants you the right to:

• Use the model for personal, academic, or commercial purposes.
• Modify the code or fine‑tune the model further.
• Distribute derived works, provided you retain the original copyright notice and include a copy of the license.

If the “unknown” tag appears in the Hugging Face UI, it is a metadata artifact; the underlying repository explicitly states Apache‑2.0. No additional attribution beyond the standard © Jonatas Grosman notice is required, but citing the model card URL is good practice.