whisper-large-v3

Technical Overview

What is this model? whisper-large-v3 is OpenAI’s latest large‑scale Whisper checkpoint for automatic speech recognition (ASR) and speech‑to‑text translation. It accepts raw audio, converts it into a mel‑spectrogram, and outputs tokenised transcriptions in more than 90 languages.

Key features & capabilities

• Multilingual support for 96 languages (English, Mandarin, Cantonese, German, French, Spanish, Russian, Korean, Japanese, Portuguese, Turkish, etc.).
• Zero‑shot generalisation across domains – no fine‑tuning required for most real‑world audio.
• Improved spectrogram resolution: 128 Mel bins (up from 80) give finer frequency detail.
• New dedicated language token for Cantonese, boosting accuracy on that dialect.
• Built‑in language detection; can be overridden for forced‑language decoding.
• Supports all Whisper decoding heuristics – temperature fallback, compression‑ratio thresholds, no‑speech detection, and timestamp generation.

Architecture highlights – The model follows the original Whisper encoder‑decoder transformer design: a 32‑layer encoder with 128‑dimensional mel‑spectrogram inputs, a 32‑layer decoder, 20‑head multi‑head attention, and a hidden size of 1280. The only architectural change in large‑v3 is the expanded mel‑frequency dimension (128 → 80) and the addition of a Cantonese language token. The model is released in torch and jax formats and is compatible with 🤗 Transformers via the automatic‑speech‑recognition pipeline.

Intended use cases – High‑accuracy transcription of podcasts, lectures, call‑center recordings, subtitles for video platforms, and speech‑to‑text translation pipelines. Its multilingual breadth makes it suitable for global content moderation, accessibility services, and research on low‑resource languages.

Benchmark Performance

Benchmarks for ASR models focus on Word Error Rate (WER) and Character Error Rate (CER) across standard corpora such as LibriSpeech, CommonVoice, and multilingual test sets. The README reports a 10 %–20 % error reduction over Whisper large‑v2 on a wide variety of languages, indicating a consistent WER drop across both high‑resource (English, Mandarin) and low‑resource (Cantonese, Swahili) datasets.

Why these benchmarks matter: WER directly translates to user‑perceived transcription quality; lower WER reduces post‑editing effort and improves downstream tasks like keyword spotting or translation. The multilingual improvements also demonstrate the model’s robustness to varied phonetic inventories and acoustic conditions.

Compared to other large‑scale ASR checkpoints (e.g., whisper‑large, whisper‑large‑v2, or commercial services like Google Speech‑to‑Text), whisper‑large‑v3 offers a measurable edge in zero‑shot performance without additional fine‑tuning, while staying fully open‑source.

Hardware Requirements

• VRAM for inference: ~15 GB for a single‑utterance batch on FP16; ~20 GB for FP32. The model can run on 24 GB GPUs (e.g., RTX 3090) with low_cpu_mem_usage=True and use_safetensors=True to reduce peak memory.
• Recommended GPU: NVIDIA RTX 3090 / RTX A6000 / AMD Radeon RX 6900 XT or any GPU with ≥ 16 GB VRAM and CUDA ≥ 11.3. For batch processing, a 32 GB GPU (e.g., RTX A6000) enables batch sizes of 8–16 without memory overflow.
• CPU requirements: A modern 8‑core CPU (Intel i7‑12700K, AMD Ryzen 7 5800X) is sufficient for loading the model and performing feature extraction. CPU‑only inference is possible but will be > 5× slower.
• Storage: The checkpoint is ~2.5 GB (safetensors) plus ~1 GB for tokenizer/feature‑extractor files. SSD storage is recommended for fast loading.
• Performance characteristics: On a RTX 3090, a 30‑second audio clip transcribes in ~0.8 seconds (FP16). Real‑time factor (RTF) ≈ 0.03, making it suitable for live streaming or batch transcription pipelines.

Use Cases

• Media & Entertainment: Automatic subtitle generation for movies, TV shows, and user‑generated video platforms (YouTube, TikTok).
• Enterprise Call‑Center Analytics: Real‑time transcription for sentiment analysis, compliance monitoring, and agent assistance.
• Accessibility: Live captioning for webinars, online courses, and public‑sector broadcasts, supporting 96 languages.
• Research & Academia: Corpus creation for low‑resource language studies, phonetics research, and speech‑to‑text translation experiments.
• Embedded Devices: When paired with a Q4KM hard‑drive, the model can be shipped pre‑loaded on edge appliances (e.g., smart speakers, translation kiosks).

Training Details

Methodology: The model was trained for 2.0 epochs over a mixed dataset of 1 M hours of human‑annotated audio and 4 M hours of pseudo‑labelled audio generated by Whisper large‑v2. The training loop employed a combination of CTC and encoder‑decoder cross‑entropy losses, with language‑token conditioning to guide multilingual output.

Datasets: Primary sources include:

• Open‑source corpora (LibriSpeech, CommonVoice, VoxPopuli, etc.) for the 1 M‑hour human‑labelled set.
• Self‑generated transcripts from Whisper large‑v2 covering diverse web‑audio, podcasts, and broadcast recordings.

Compute: Training leveraged a cluster of 64 × NVIDIA A100‑40 GB GPUs, using mixed‑precision (FP16) and gradient checkpointing to fit the 1.5 B‑parameter model. Estimated total compute ≈ 1 M GPU‑hours.

Fine‑tuning: The checkpoint is fully compatible with AutoModelForSpeechSeq2Seq and can be fine‑tuned on domain‑specific data (e.g., medical dictation) using the standard Hugging Face Trainer API. Because the model already generalises well, modest fine‑tuning (≤ 10 k steps) often yields noticeable gains.

Related Papers

The core research behind Whisper is documented in Robust Speech Recognition via Large‑Scale Weak Supervision (arXiv:2212.04356) by Alec Radford et al., OpenAI. This paper introduces the weak‑supervision training pipeline, the multilingual token set, and the zero‑shot evaluation methodology that underpin whisper‑large‑v3.

Key contributions that directly affect large‑v3 include:

• Training on > 5 M hours of audio (1 M human‑labelled + 4 M pseudo‑labelled).
• Use of a unified encoder‑decoder transformer architecture for both ASR and speech translation.
• Demonstrated robustness to background noise, accents, and domain shift.

Licensing Information

The model card lists the Apache‑2.0 license for the checkpoint files, while the overall license: unknown flag reflects that some auxiliary assets (e.g., dataset scripts) may have separate terms. Under Apache‑2.0 you may:

• Use the model commercially, embed it in SaaS products, or ship it with hardware.
• Modify the weights or code, provided you retain the original copyright notice.
• Distribute derivative works under a different license, as long as you include a copy of the Apache‑2.0 license.

If you plan to redistribute the model (e.g., on a hard‑drive product), you must:

• Provide proper attribution to OpenAI and the Hugging Face repository.
• Include a copy of the Apache‑2.0 license text.
• Ensure any third‑party data used for fine‑tuning respects its own licenses.