whisper-medium

Technical Overview

Whisper‑Medium is a large‑scale, multilingual automatic‑speech‑recognition (ASR) model released by OpenAI. It is a transformer‑based encoder‑decoder that converts raw audio into a sequence of text tokens, supporting more than 90 languages. The model is part of the Whisper family, trained on 680 k hours of weakly supervised speech data, which gives it strong generalisation across domains, accents, and recording conditions without any fine‑tuning.

Key features & capabilities include:

• Multilingual transcription (English, Mandarin, German, Spanish, Russian, Korean, French, Japanese, Portuguese, Turkish, Polish, Catalan, Dutch, Arabic, Swedish, Italian, Indonesian, Hindi, Finnish, Vietnamese, Hebrew, Ukrainian, Greek, Malay, Czech, Romanian, Danish, Hungarian, Tamil, Norwegian, Thai, Urdu, Croatian, Bulgarian, Lithuanian, Latin, Malayalam, etc.)
• Robustness to background noise, reverberation and low‑quality recordings thanks to the massive, diverse training set.
• Support for both speech‑to‑text and speech‑to‑translation (the multilingual checkpoints can output translations in a target language).
• End‑to‑end inference with a single WhisperProcessor that handles audio resampling, log‑Mel spectrogram extraction and tokenisation.

Architecture highlights:

• Transformer encoder‑decoder with 32 attention layers (12 encoder, 20 decoder) and a total of ~769 M parameters.
• Input is a 30‑second log‑Mel spectrogram (80 mel bins, 30 s stride) – the model can process longer audio by sliding‑window chunking.
• Joint CTC‑decoder training encourages alignment while preserving the flexibility of a seq‑2‑seq decoder.
• Positional encoding is relative, which improves handling of variable‑length inputs.

Intended use cases range from real‑time captioning, podcast transcription, and call‑center analytics to low‑resource language documentation and multilingual subtitle generation.

Benchmark Performance

For ASR models, the most common benchmark is Word Error Rate (WER) on clean and noisy test splits of LibriSpeech, as well as language‑specific WER on Mozilla Common Voice. Whisper‑Medium’s README reports the following results:

• LibriSpeech (clean) test WER: 2.9 %
• LibriSpeech (other) test WER: 5.9 %
• Common Voice 11.0 (Hindi) test WER: 53.87 %

These numbers illustrate Whisper‑Medium’s ability to stay under 3 % WER on high‑quality English speech while still delivering reasonable performance on far noisier or non‑English data. Compared with the smaller whisper‑small (≈244 M parameters) which typically scores ~4 % WER on clean LibriSpeech, Whisper‑Medium offers a clear accuracy boost at a modest increase in compute cost. It also outperforms many open‑source ASR baselines (e.g., wav2vec‑2.0 large) on multilingual benchmarks, making it a strong candidate for production‑grade transcription pipelines.

Hardware Requirements

Whisper‑Medium’s 769 M parameters translate into the following practical hardware recommendations:

• GPU VRAM for inference: 8 GB is the absolute minimum for a single 30 s chunk; 12 GB+ (e.g., RTX 3060 12 GB, RTX A5000) enables batch processing and lower latency.
• Recommended GPU: NVIDIA RTX 3080 10 GB or AMD RX 6800 16 GB for real‑time or near‑real‑time transcription of 30 s audio.
• CPU: Modern multi‑core CPUs (Intel i7‑9700K, AMD Ryzen 7 3700X) can run the model in “CPU‑only” mode, but expect 5‑10× slower throughput.
• Storage: The model checkpoint (including safetensors) occupies roughly 2.5 GB. Keep at least 5 GB free for the model, audio cache, and temporary spectrogram files.
• Performance characteristics: On a RTX 3080, Whisper‑Medium transcribes a 30 s audio clip in ~0.6 s (≈50× real‑time). Latency scales linearly with audio length.

Use Cases

Whisper‑Medium shines in scenarios where high‑accuracy transcription across many languages is needed without a costly fine‑tuning stage. Typical applications include:

• Media & Entertainment: Automatic subtitle generation for movies, TV shows, and streaming platforms in multilingual markets.
• Customer Support: Real‑time call‑center transcription and sentiment analysis for agents handling global customers.
• Education & Research: Transcribing lectures, podcasts, and oral histories in under‑represented languages.
• Accessibility: Live captioning for webinars, conferences, and public‑sector broadcasts.
• Productivity Tools: Voice‑to‑text features in note‑taking apps, meeting‑recording software, and virtual assistants.

Integration is straightforward via the Hugging Face transformers library and the WhisperProcessor. The model can be served through REST APIs, embedded in Python services, or exported to ONNX/TensorRT for low‑latency edge deployment.

Training Details

Whisper‑Medium was trained on a massive, publicly available audio corpus consisting of 680 k hours of speech from sources such as LibriVox, Common Voice, and multilingual YouTube recordings. The data were weakly supervised: transcripts were automatically generated using existing ASR systems and then filtered for quality.

Methodology:

• Sequence‑to‑sequence transformer with 32 layers (12 encoder, 20 decoder).
• Joint CTC and decoder loss to encourage alignment while preserving language modeling power.
• Training used mixed‑precision (FP16) on a cluster of 64 × NVIDIA A100 GPUs for roughly 2 weeks, consuming an estimated 1 M GPU‑hours.
• Data augmentation included random speed perturbation (0.9‑1.1×), volume scaling, and additive background noise.

The model is released as a pre‑trained checkpoint only; fine‑tuning is optional but not required for most downstream tasks. Users can adapt Whisper‑Medium to domain‑specific vocabularies via a lightweight language‑model head or by employing Hugging Face Trainer with a small amount of labeled audio.

Related Papers

The core research behind Whisper‑Medium is described in the following peer‑reviewed work:

• Robust Speech Recognition via Large‑Scale Weak Supervision – Radford et al., 2022. This paper introduces the Whisper architecture, the 680 k hour weakly supervised dataset, and the multilingual training regime.

Subsequent analyses (e.g., “Whisper: A General‑Purpose Speech Model” by various community members) have explored its zero‑shot translation abilities and robustness to noise, further cementing its status as a benchmark‑setting ASR system.

Licensing Information

The model is released under the Apache‑2.0 license (the README lists license: apache-2.0). This permissive license grants:

• Free use for both research and commercial purposes.
• The right to modify, distribute, and create derivative works.
• Patent‑grant protection for contributors.

Restrictions are minimal:

• Any redistribution must retain the original copyright notice and license text.
• Trademark use (e.g., “OpenAI”) requires separate permission.

For commercial deployments, you may embed the model in SaaS products, on‑device applications, or edge devices, provided you honour the attribution clause (e.g., “© OpenAI, licensed under Apache‑2.0”). No royalty fees are required.