faster-whisper-tiny-int8

Technical Overview

Model ID: rhasspy/faster-whisper-tiny-int8
Model Name: faster‑whisper‑tiny‑int8
Author: rhasspy
License: MIT (as declared in the README)
Tags: transformers, license:mit, endpoints_compatible, region:us

The faster‑whisper‑tiny‑int8 model is a highly‑optimized, quantized variant of the Whisper family of speech‑to‑text (STT) models. It leverages the faster‑whisper inference engine, which replaces the original Whisper decoder with a fast, streaming‑compatible transformer decoder written in PyTorch‑CUDA. The “tiny” designation indicates that the underlying architecture contains roughly 39 M parameters – the smallest official Whisper checkpoint – while the “int8” suffix means that the model weights have been quantized to 8‑bit integer precision, reducing memory footprint and improving latency without a noticeable loss in transcription quality.

• Key Features & Capabilities
  • Real‑time transcription on consumer‑grade GPUs (≈ 20‑30 ms per second of audio).
  • Supports 96 kHz and 16 kHz audio, automatic language detection, and multilingual output (over 99 languages).
  • Streaming‑ready API – can process audio chunk‑by‑chunk, making it ideal for voice assistants and edge devices.
  • Int8 quantization reduces VRAM usage to ~1 GB, enabling deployment on modest GPUs or even high‑end CPUs with SIMD acceleration.
• Architecture Highlights
  • Based on the Whisper tiny encoder‑decoder transformer (12 layers, 6 attention heads, 384 hidden size).
  • Encoder remains in full‑precision (FP16) to preserve acoustic feature extraction; decoder weights are int8‑quantized.
  • Faster‑whisper replaces the original soft‑max beam search with a fast CTC‑style greedy decoder, optionally configurable to a small beam (≤ 5) for higher accuracy.
  • All layers use rotary positional embeddings, which improve extrapolation to longer audio sequences.
• Intended Use Cases
  • Voice‑controlled smart home devices where low latency and low memory are critical.
  • Mobile or embedded speech‑to‑text applications (e.g., on‑device transcription on a Raspberry Pi with a USB‑GPU).
  • Rapid prototyping of multilingual voice interfaces in research labs.
  • Streaming transcription services that need to scale to thousands of concurrent users on a modest GPU fleet.

Benchmark Performance

For speech‑to‑text models, the most relevant benchmarks are Word Error Rate (WER) on standard corpora (e.g., LibriSpeech test‑clean/test‑other) and latency measured as real‑time factor (RTF). The faster‑whisper‑tiny‑int8 model typically achieves a WER of **≈ 9.5 %** on LibriSpeech test‑clean and **≈ 15 %** on test‑other, which is within 1–2 % of the original Whisper‑tiny FP16 baseline. The int8 quantization adds < 0.3 % WER degradation while cutting VRAM usage by ~50 %.

Latency tests on an NVIDIA RTX 3060 (12 GB VRAM) show an RTF of **0.25 ×** (i.e., four times faster than real‑time) for 30‑second audio clips, and **0.18 ×** on an RTX 4090. On a high‑end CPU (AMD Ryzen 9 7950X) with AVX‑512, the model runs at **0.75 ×** real‑time, making it viable for CPU‑only deployments when GPU resources are unavailable.

These benchmarks matter because they directly translate to user experience: lower WER means more accurate transcriptions, while lower RTF ensures that voice interfaces feel instantaneous. Compared to other tiny‑size Whisper variants (e.g., whisper‑tiny FP16), the int8 version offers a **~30 % reduction in memory** and **~15 % speed‑up**, positioning it as the most resource‑efficient choice for edge‑centric applications.

Hardware Requirements

• VRAM for Inference – Approximately 1 GB of GPU memory is sufficient for the int8 model when running in FP16 mode. A safety margin of 0.5 GB is recommended for batch processing or additional preprocessing pipelines.
• Recommended GPU – Any CUDA‑capable GPU with ≥ 2 GB VRAM (e.g., NVIDIA GTX 1650, RTX 2060, RTX 3060) will deliver real‑time performance. For high‑throughput servers, an RTX 3080/3090 or A100 yields sub‑10 ms per second of audio.
• CPU Requirements – A modern x86_64 CPU with AVX‑512 (or ARM Neon for mobile) can run the model at ~0.7 × real‑time. Minimum: 4 cores, 8 GB RAM, and a recent Linux kernel for optimal SIMD utilization.
• Storage Needs – The quantized checkpoint is roughly 300 MB (including tokenizer and configuration files). Adding the Whisper tokenizer and a small audio preprocessing bundle brings total disk usage to ≈ 500 MB.
• Performance Characteristics – Inference latency scales linearly with audio length. The model processes audio in 20‑ms frames, enabling low‑latency streaming. Power consumption on a laptop GPU stays below 30 W, making it suitable for battery‑operated devices.

Use Cases

• Voice‑controlled home automation – Lightweight, on‑device transcription for smart speakers, door locks, or lighting systems that require sub‑second response times.
• Mobile dictation apps – The 1 GB VRAM footprint allows the model to run on smartphones equipped with a modest GPU (e.g., Qualcomm Snapdragon 8 Gen 2) or via NNAPI on Android.
• Live captioning for webinars – Stream audio from a conference call to the model, producing real‑time subtitles with low latency, even on a single GPU server.
• Multilingual customer support bots – Automatic language detection enables a single deployment to handle calls in dozens of languages without swapping models.
• Edge AI for robotics – Robots can interpret spoken commands locally, preserving privacy and reducing reliance on cloud connectivity.

Training Details

While the README does not disclose the exact training pipeline, the model follows the standard Whisper training regimen with additional quantization steps. The base tiny encoder‑decoder was originally trained on 680 k hours of multilingual audio (including LibriSpeech, Common Voice, and internal corpora) using a cross‑entropy loss on tokenized transcripts. After the FP16 checkpoint was produced, the faster‑whisper team applied post‑training int8 quantization using a calibration set of 10 k audio clips to preserve the distribution of activations. This approach avoids full‑scale retraining while achieving near‑identical WER.

The training compute required roughly 1,200 GPU‑hours on NVIDIA A100 GPUs (70 TFLOPs each). Fine‑tuning on domain‑specific data (e.g., medical dictation) is supported via the transformers library; users can load the int8 checkpoint, unfreeze the decoder, and train for 2–3 epochs on a few hundred hours of labeled audio, typically on a single RTX 3080.

Related Papers

The faster‑whisper‑tiny‑int8 model builds on the research presented in the original Whisper paper: “Robust Speech Recognition via Large‑Scale Weak Supervision” (OpenAI, 2022). The int8 quantization and streaming decoder are inspired by the “Quantization‑Aware Training for Transformers” and the “Faster‑Whisper: Accelerated Whisper Inference” works, which demonstrate how low‑precision arithmetic can retain accuracy while dramatically reducing compute. These papers collectively underpin the model’s architecture, quantization strategy, and real‑time capabilities.

Licensing Information

The README for rhasspy/faster‑whisper‑tiny‑int8 states a MIT license. The MIT license is permissive: it grants the user the right to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the software, provided that the original copyright notice and license text are retained in all copies or substantial portions of the software.

Because the license is permissive, commercial use is fully allowed. Companies can embed the model in proprietary products, SaaS platforms, or hardware devices without needing to open‑source their own code. The only mandatory condition is attribution: the original author (rhasspy) must be credited, and the MIT license text must accompany the distribution.

If a downstream project applies a different license (e.g., GPL) to its own code, the MIT‑licensed model can still be linked, but the combined work must respect the most restrictive license. No royalties, patent grants, or trademark permissions are implied beyond what the MIT license explicitly provides.