whisper-tiny

Technical Overview

What is this model? Whisper‑Tiny is the smallest checkpoint in OpenAI’s Whisper family, a transformer‑based encoder‑decoder system for automatic speech recognition (ASR) and speech translation. With only ≈ 39 million parameters, it is designed for low‑latency, on‑device, or edge‑deployment scenarios while still benefiting from the same massive weak‑supervision training that powers the larger Whisper models.

Key features and capabilities

• Supports 99 languages (including English, Mandarin, Hindi, Arabic, Russian, and many more) directly out‑of‑the‑box.
• End‑to‑end transcription without the need for language‑specific fine‑tuning.
• Built‑in language detection – the model can infer the spoken language before transcribing.
• Works with the WhisperProcessor to convert raw audio into log‑Mel spectrograms, simplifying preprocessing.
• Small memory footprint enables inference on consumer‑grade GPUs, recent CPUs, and even some mobile SoCs.

Architecture highlights

• Transformer encoder‑decoder with 6 encoder layers and 4 decoder layers.
• Each encoder layer contains a multi‑head self‑attention block (4 heads) and a feed‑forward network (size = 1536).
• Positional embeddings are learned, and the model processes 30 ms frames at a 16 kHz sampling rate.
• Joint CTC‑style loss for speech recognition and an optional translation head for multilingual tasks.

Intended use cases

• Real‑time captioning for video conferencing or live streaming where latency and GPU memory are limited.
• Offline transcription on laptops, tablets, or edge devices such as smart speakers.
• Rapid prototyping of multilingual ASR pipelines without maintaining separate language‑specific models.
• Data‑augmentation for downstream NLP tasks (e.g., generating subtitles for low‑resource languages).

Benchmark Performance

Whisper‑Tiny has been evaluated on two classic ASR benchmarks: LibriSpeech (clean & other) and Mozilla Common Voice 11.0 (Hindi). The results are reported as Word Error Rate (WER), the standard metric for speech transcription quality.

• LibriSpeech (clean) – Test WER: 7.54 %
• LibriSpeech (other) – Test WER: 17.15 %
• Common Voice 11.0 (Hindi) – Test WER: 141 %

These benchmarks matter because LibriSpeech represents clean, high‑fidelity English speech, while the “other” split introduces background noise, speaker variation, and recording artifacts. The Hindi Common Voice result highlights the challenges of low‑resource languages for a tiny model. Compared with larger Whisper checkpoints (e.g., whisper‑base with ~74 M parameters achieving ~5 % WER on clean LibriSpeech), Whisper‑Tiny trades a modest increase in error for a dramatic reduction in compute and memory requirements, making it competitive for latency‑critical applications.

Hardware Requirements

VRAM for inference – Whisper‑Tiny comfortably runs on a single GPU with as little as 2 GB of VRAM when using half‑precision (FP16) tensors. For full‑precision (FP32) inference, allocate 4 GB to avoid out‑of‑memory errors on long audio clips.

Recommended GPU – Any modern NVIDIA GPU with CUDA support (e.g., RTX 3050, GTX 1660 Ti, or the newer RTX 40‑series) will deliver real‑time transcription (< 30 ms per second of audio). For CPU‑only deployments, a recent 8‑core Xeon or AMD Ryzen processor can achieve ~1× real‑time speed when using the optimized torch.compile path.

CPU & storage – The model file (including tokenizer and config) is ≈ 300 MB. A solid‑state drive (SSD) is recommended to keep load times under a second. The WhisperProcessor adds another ~150 MB for feature extraction assets.

Performance characteristics – Inference latency scales linearly with audio length. On a RTX 3060 (12 GB VRAM) Whisper‑Tiny processes a 10‑second clip in ~0.2 seconds (FP16) and ~0.35 seconds (FP32). Memory usage stays below 1 GB, leaving head for batch processing or simultaneous model instances.

Use Cases

Whisper‑Tiny shines in scenarios where latency, memory, and deployment cost outweigh the need for the highest possible transcription accuracy.

• Live captioning for webinars and virtual classrooms – Real‑time subtitles for multilingual audiences without a heavy GPU server.
• Smart‑home voice assistants – On‑device command recognition that respects user privacy by avoiding cloud round‑trips.
• Mobile journalism and field reporting – Quick transcription of interview recordings on phones or tablets.
• Accessibility tools – Generating subtitles for video platforms that need to process thousands of videos per day on modest hardware.
• Data‑labeling pipelines – Automatic pre‑transcription of large audio corpora before human correction, reducing labeling costs.

Training Details

Whisper‑Tiny was trained on a massive weakly‑supervised dataset comprising roughly 680 k hours of audio‑text pairs harvested from the internet. The data spans 99 languages and includes both clean speech and noisy, real‑world recordings.

• Model size: 39 M parameters, 6 encoder layers, 4 decoder layers.
• Training objective: A combination of CTC loss for alignment‑free transcription and a cross‑entropy loss for language‑aware decoding.
• Optimization: AdamW optimizer with a cosine learning‑rate schedule, trained on mixed‑precision (FP16) on a cluster of NVIDIA A100 GPUs (8 × 40 GB VRAM) for roughly 2 weeks of wall‑clock time.
• Data augmentation: Random time‑stretching, volume perturbation, and SpecAugment to improve robustness to noise and speaker variability.
• Fine‑tuning: While the base checkpoint is ready for zero‑shot use, the Hugging Face Trainer API can be employed to fine‑tune Whisper‑Tiny on domain‑specific corpora (e.g., medical dictation) with as little as a few hundred labeled minutes.

Related Papers

The core research behind Whisper‑Tiny is documented in the paper “Robust Speech Recognition via Large‑Scale Weak Supervision” (arXiv:2212.04356) by Alec Radford et al., OpenAI. This work introduced the weak‑supervision training regime and the multilingual encoder‑decoder architecture that powers all Whisper checkpoints, including the tiny variant.

Additional background can be found in the original Whisper code repository: github.com/openai/whisper, which details the data preprocessing pipeline, the CTC‑plus‑language‑model loss, and the evaluation methodology across 99 languages.

Licensing Information

The model card lists the license as Apache‑2.0. This permissive open‑source license grants you broad freedoms:

• Use, modify, and distribute the model and its weights for both research and commercial purposes.
• Combine Whisper‑Tiny with proprietary code or other licenses without viral effects.
• Patent protection for contributors – you receive a royalty‑free license to any patents covering the model.

Commercial usage – Fully allowed under Apache‑2.0, provided you retain the original copyright notice and a copy of the license in any distribution. No “unknown” restrictions apply because the README clarifies the Apache‑2.0 status.

Attribution – When redistributing the model (e.g., as part of a SaaS offering or a packaged SDK), include a statement such as: “Whisper‑Tiny, © OpenAI, licensed under the Apache‑2.0 License.” A link to the original Hugging Face model card (https://huggingface.co/openai/whisper-tiny) satisfies the attribution requirement.