wav2vec2-large-xlsr-53-persian

Technical Overview

The jonatasgrosman/wav2vec2-large-xlsr-53-persian model is a fine‑tuned version of Facebook’s wav2vec2‑large‑xlsr‑53 adapted for Persian (Farsi) automatic speech recognition (ASR). It operates directly on raw audio waveforms sampled at 16 kHz and outputs a sequence of characters using a CTC (Connectionist Temporal Classification) head. The model is built on the XLSR‑53 multilingual self‑supervised backbone, which was pre‑trained on 53 languages and then fine‑tuned on the Persian split of the Common Voice 6.1 dataset.

Key features and capabilities include:

• Language support: Persian (fa) with a vocabulary tailored to the Persian script.
• End‑to‑end inference: No external language model required; the processor handles tokenisation, feature extraction and decoding.
• High‑accuracy ASR: Achieves 30.12 % Word Error Rate (WER) and 7.37 % Character Error Rate (CER) on the Common Voice Persian test set.
• Compatibility: Works with 🤗 Transformers, PyTorch, JAX, and the Huggingsound library for rapid prototyping.
• Deployable: Ready for Azure deployment (tagged deploy:azure) and can be served via Hugging Face Inference API.

Architecture highlights:

• Base encoder: 24 transformer blocks, 1024 hidden size, 16 attention heads – the “large” configuration of wav2vec2.
• Self‑supervised pre‑training on 53 languages (XLSR‑53) provides robust acoustic representations that transfer well to low‑resource languages.
• CTC classification head fine‑tuned on Persian transcriptions, mapping encoder outputs to a 32‑character Persian alphabet plus special tokens.

Intended use cases span any application that requires converting Persian speech to text: voice assistants, transcription services, subtitle generation, and accessibility tools for the hearing impaired.

Benchmark Performance

For Persian ASR, the most informative benchmarks are Word Error Rate (WER) and Character Error Rate (CER), which directly reflect transcription quality at the word and character levels respectively. The model’s evaluation on the Common Voice Persian test split reports:

• Test WER: 30.12 %
• Test CER: 7.37 %

These numbers are significant because they place the model among the top open‑source Persian ASR solutions, especially given the limited amount of high‑quality Persian speech data. A lower CER indicates that the model handles the script’s complex orthography well, while the WER demonstrates overall sentence‑level accuracy. Compared to the base facebook/wav2vec2-large-xlsr-53 model (which typically yields >40 % WER on Persian without fine‑tuning), the fine‑tuned version offers a clear performance gain, making it suitable for production‑grade deployments.

Hardware Requirements

The model’s “large” wav2vec2 backbone demands modest yet specific hardware for efficient inference.

• VRAM: Approximately 4 GB of GPU memory is sufficient for batch‑size = 1 inference at 16 kHz. Larger batches or mixed‑precision (FP16) can be run on 8 GB GPUs.
• Recommended GPUs: NVIDIA RTX 3060/3070, Tesla T4, or any GPU supporting CUDA 11+ with at least 6 GB VRAM.
• CPU: A modern multi‑core CPU (e.g., Intel i7‑9700K or AMD Ryzen 7 3700X) can run inference on‑device, though latency will be higher than GPU.
• Storage: Model checkpoint size is ~1.2 GB (including processor and tokenizer). Allocate at least 2 GB of free disk space for the model and auxiliary files.
• Performance: On a RTX 3060, a 10‑second audio clip is transcribed in ~0.15 seconds (≈66 × real‑time). CPU‑only inference on a 4‑core CPU typically runs at ~0.8 × real‑time.

Use Cases

The Persian wav2vec2 model unlocks a wide range of applications where accurate speech‑to‑text conversion is essential.

• Voice‑enabled assistants: Power Persian‑language smart speakers, mobile assistants, and chatbot back‑ends.
• Media transcription: Generate subtitles for Persian podcasts, YouTube videos, and broadcast news.
• Accessibility tools: Provide real‑time captioning for the hearing impaired in classrooms, conferences, and public services.
• Customer support analytics: Transcribe call‑center recordings for sentiment analysis and quality monitoring.
• Legal and medical dictation: Convert spoken notes into written records while preserving medical terminology.

Integration is straightforward via the Hugging Face transformers library, the huggingsound wrapper, or the model’s ONNX export for edge deployment. The model’s compatibility with Azure also enables scalable cloud‑based transcription pipelines.

Training Details

The model was fine‑tuned on the Persian split of Common Voice 6.1. Training used the train and validation splits, with the following configuration:

• Base model: facebook/wav2vec2-large-xlsr-53 (pre‑trained on 53 languages).
• Dataset: ~200 hours of Persian speech recordings, each paired with a normalized transcription.
• Training script: wav2vec2‑sprint (open‑source).
• Compute: Fine‑tuning performed on OVHcloud GPU instances (NVIDIA Tesla V100, 16 GB VRAM) for approximately 12 hours.
• Optimization: AdamW optimizer, learning rate warm‑up to 1e‑4, CTC loss.
• Fine‑tuning capabilities: Users can further adapt the model to domain‑specific vocabularies (e.g., medical or legal Persian) by continuing training on a smaller, labelled dataset.

All audio inputs must be resampled to 16 kHz before processing, as the model’s feature extractor expects this sampling rate.

Related Papers

The foundation of this model rests on two seminal works:

• wav2vec 2.0: A Framework for Self‑Supervised Learning of Speech Representations – Introduces the self‑supervised pre‑training methodology that underpins the XLSR‑53 backbone.
• XLSR‑53: Cross‑Lingual Speech Representation Learning at Scale – Extends wav2vec 2.0 to 53 languages, providing the multilingual acoustic model later fine‑tuned for Persian.

The fine‑tuning script used for this model is hosted in the wav2vec2‑sprint repository, which follows the training recipes described in the above papers.

Licensing Information

The repository tags indicate an Apache 2.0 license, even though the model card lists the license as “unknown”. Under Apache 2.0, users receive:

• Broad permission to use, modify, and distribute the model for both commercial and non‑commercial purposes.
• No warranty; the model is provided “as‑is”.
• Obligation to retain the original copyright notice and provide a copy of the license in any redistributed version.

Because Apache 2.0 is permissive, you can integrate the model into commercial products, SaaS platforms, or embedded devices without paying royalties. The only restriction is proper attribution to the original author, jonatasgrosman, and to the underlying facebook/wav2vec2-large-xlsr-53 model. If you plan to redistribute the model binaries, include the license file and a clear notice of any modifications you have made.