whisperkit-coreml

Technical Overview

What is this model? whisperkit‑coreml is a pre‑converted CoreML version of the Whisper speech‑to‑text architecture, optimized for Apple Silicon devices (M‑series CPUs/GPUs and iOS/iPadOS). It brings the power of OpenAI’s Whisper model onto the edge, allowing developers to run automatic‑speech‑recognition (ASR) entirely on‑device without a network connection.

Key features and capabilities

• Fully on‑device inference – no cloud latency, privacy‑first processing.
• Quantized weights (8‑bit integer) for a dramatically reduced memory footprint while preserving Whisper‑level accuracy.
• Real‑time streaming API that can process audio chunks as short as 20 ms.
• Custom vocabulary injection and speaker diarization support via the Argmax SDK.
• Compatibility with iOS 15+, macOS 12+, and visionOS – all via the native CoreML runtime.

Architecture highlights The underlying backbone mirrors Whisper’s transformer encoder‑decoder, but the CoreML conversion applies:

• Operator fusion to collapse multi‑stage attention into a single Metal‑accelerated kernel.
• Static graph pruning that removes the language‑model head for models that only need transcription.
• Post‑training quantization to 8‑bit, which reduces the model size from ~1.5 GB (float‑16) to ~350 MB (int8) while keeping word‑error‑rate (WER) within 0.3 % of the original.

Intended use cases WhisperKit‑CoreML is designed for:

• Voice assistants on iPhone, iPad, and Apple Watch that require instant, offline transcription.
• Live captioning for video‑conferencing apps where privacy and latency are critical.
• Field‑recording tools for journalists and researchers who need on‑device transcription in remote locations.
• Embedded speech‑driven controls in macOS desktop applications.

Benchmark Performance

Relevant benchmarks For on‑device ASR, the most informative metrics are:

• Word‑Error‑Rate (WER) – a direct measure of transcription accuracy.
• Real‑time factor (RTF) – the ratio of processing time to audio duration; an RTF < 1.0 indicates true real‑time performance.
• Throughput (tokens / second) – useful for streaming pipelines.

Performance from the README & public tests The WhisperKit paper (ICML 2025) reports an average WER of 5.2 % on the LibriSpeech test‑clean set using the quantized CoreML model on an M2‑Pro MacBook. The real‑time factor on the same hardware is 0.68, meaning the model transcribes 1 second of audio in ~0.68 seconds. On an iPhone 14 Pro, the RTF rises to 0.92, still comfortably below the 1.0 threshold for live streaming.

Why these benchmarks matter Low WER ensures transcription quality, while a sub‑1.0 RTF guarantees that users experience no lag, a prerequisite for voice‑controlled UI and live captioning. Compared with the original Whisper‑small (float‑16) running on GPU, WhisperKit‑CoreML delivers a 2‑3× speedup on Apple Silicon with only a marginal accuracy loss.

Hardware Requirements

VRAM / memory footprint The quantized CoreML model occupies ~350 MB of RAM at runtime. Apple’s Metal framework allocates an additional 150 MB for intermediate tensors, so a device with at least 1 GB of free GPU memory (e.g., M1, M2, M3 chips) is sufficient.

Recommended GPU specifications For optimal latency, use:

• Apple M2‑Pro / M2‑Max / M3‑Pro with ≥ 16 GPU cores.
• iPhone 14 Pro or later (A16 Bionic) for mobile deployment.

CPU requirements The CoreML runtime can fall back to the CPU if the GPU is unavailable, but real‑time performance degrades to an RTF of ~1.6 on an Intel i7‑12700H. Therefore, a modern Apple Silicon CPU (≥ 8 performance cores) is recommended.

Storage needs The model file (CoreML .mlmodelc bundle) is ~350 MB. Including the Argmax SDK and sample audio data, a typical app bundle will be under 500 MB, well within the 5 GB limit for iOS app store submissions.

Performance characteristics On an M2‑Max, WhisperKit‑CoreML processes 30 seconds of audio in ~20 seconds (RTF ≈ 0.66) with a peak power draw of ~2 W, making it suitable for battery‑powered devices.

Use Cases

Primary intended applications WhisperKit‑CoreML shines in scenarios where latency, privacy, and offline capability are non‑negotiable:

• Voice assistants on iPhone, iPad, Apple Watch, and Mac that must respond instantly without sending audio to the cloud.
• Live captioning for video‑conferencing or streaming services that need on‑device transcription to comply with accessibility regulations.
• Field recording tools for journalists, law‑enforcement, or researchers operating in low‑connectivity environments.
• Smart home controllers that run on Apple TV or HomePod, providing local speech command processing.

Real‑world examples

• Apple’s “Voice MOver” feature could integrate WhisperKit‑CoreML to provide instantaneous captions on iOS devices.
• A medical‑note‑taking app could transcribe doctor‑patient conversations offline, preserving patient confidentiality.
• Educational platforms could embed WhisperKit‑CoreML into iPad‑based language‑learning apps for real‑time pronunciation feedback.

Integration possibilities The model is distributed as a CoreML bundle, which can be loaded with MLModel(contentsOf:) in Swift. Combined with the Argmax SDK, developers gain:

• Streaming audio pipelines via AVAudioEngine.
• Custom vocabulary injection through MLDictionaryFeatureProvider.
• Speaker diarization callbacks for multi‑speaker scenarios.

Training Details

Training methodology WhisperKit‑CoreML does not train a new model from scratch; instead it fine‑tunes the publicly released Whisper‑small (or Whisper‑base) checkpoint on a curated Apple‑specific dataset to improve on‑device performance. The fine‑tuning steps include:

• Data augmentation with Apple‑device microphone profiles.
• Post‑training quantization using TensorFlow Lite’s integer‑only quantizer, then conversion to CoreML via coremltools.
• Operator fusion to leverage Metal Performance Shaders.

Datasets used The fine‑tuning set combines:

• LibriSpeech (960 h) and CommonVoice (2 k h) for baseline language coverage.
• Apple‑internal “VoiceKit” recordings (≈ 200 h) captured on iPhone and Mac microphones to align acoustic characteristics.

Compute requirements Fine‑tuning was performed on a cluster of 8 × NVIDIA A100 GPUs for roughly 48 hours, consuming ~1 M GPU‑hours. The quantization step is lightweight and runs on a single GPU in under 2 hours.

Fine‑tuning capabilities forstrong> Users can further adapt the model via the Argmax SDK:

• Custom vocabulary injection (up to 5 k tokens).
• Domain‑specific fine‑tuning using a small “adapter” layer (≈ 10 M parameters).