mobilevit-small

Technical Overview

Model ID: apple/mobilevit-small – a compact Vision‑Transformer variant called MobileViT‑S. It is a lightweight, mobile‑friendly convolution‑transformer hybrid that delivers high‑quality image classification while keeping the parameter count low (≈ 5.6 M). Trained on the ImageNet‑1k dataset at a 256 × 256 resolution, the model can be used out‑of‑the‑box for any task that requires assigning one of 1 000 ImageNet class labels to an input image.

Key features and capabilities

• Combines MobileNetV2‑style inverted‑residual blocks with a global‑attention MobileViT block, achieving a strong balance between local convolutional processing and long‑range transformer context.
• No positional embeddings are required, simplifying the preprocessing pipeline.
• Supports both PyTorch and TensorFlow (via the transformers library) and can be exported to CoreML for on‑device inference.
• Multi‑scale training (160 – 320 px) yields robustness to varying image sizes at inference.

Architecture highlights

• Initial stem: 3×3 convolution → batch‑norm → ReLU, followed by a MobileNetV2‑style inverted‑residual block.
• MobileViT block: patches are extracted, flattened, processed by a lightweight transformer encoder (4 attention heads, 2 transformer layers), then reshaped back into a spatial feature map.
• Three stages of alternating convolutional and MobileViT blocks, ending with a global average‑pool and a fully‑connected classifier.
• Overall parameter budget: ~5.6 M, making it suitable for edge devices and real‑time applications.

Intended use cases

• On‑device image classification for smartphones, tablets, and embedded vision systems.
• Fast prototyping of vision pipelines where latency and memory are critical constraints.
• Baseline model for fine‑tuning on domain‑specific datasets (e.g., medical imaging, retail product recognition).

Benchmark Performance

MobileViT‑S was evaluated on the ImageNet‑1k validation set at 256 × 256 resolution. The official results are:

These metrics matter because ImageNet is the de‑facto benchmark for general‑purpose image classification. A top‑1 accuracy of 78 % places MobileViT‑S on par with other mobile‑oriented architectures such as EfficientNet‑B0 while using fewer FLOPs, making it an attractive choice for latency‑sensitive deployments. Compared to the smaller MobileViT‑XS (74.8 % top‑1) and MobileViT‑XXS (69.0 % top‑1), the “small” variant offers a clear accuracy boost with a modest increase in model size.

Hardware Requirements

VRAM for inference – The model occupies roughly 22 MB of GPU memory when loaded in FP32; FP16 reduces this to ~12 MB. A GPU with ≥ 2 GB VRAM (e.g., NVIDIA GTX 1650) is more than sufficient for a single‑image batch.

Recommended GPU – For batch inference or real‑time video streams, a mid‑range GPU such as the NVIDIA RTX 3060 (12 GB VRAM) or Apple M‑series (Neural Engine) provides ample headroom.

CPU requirements – On CPU‑only systems, inference runs comfortably on modern x86‑64 CPUs with ≥ 8 GB RAM. Expect ~30‑40 ms latency per 256 × 256 image on a 2.6 GHz 8‑core processor using PyTorch’s TorchScript or ONNX Runtime.

Storage – The model files (config, weights, tokenizer) total ~30 MB. Including the optional CoreML export adds another ~10 MB. A standard SSD or even high‑capacity HDD is adequate.

Performance characteristics – MobileViT‑S delivers ~150 M FLOPs per forward pass, translating to > 200 FPS on a RTX 3060 at 256 × 256 resolution, and > 30 FPS on an Apple M1‑Pro in CoreML mode.

Use Cases

MobileViT‑S shines in scenarios where low latency, small memory footprint, and high accuracy are simultaneously required.

• Mobile photography apps: Real‑time scene classification to suggest filters or auto‑enhance settings.
• Industrial edge devices: Quick defect detection on production lines using embedded GPUs or NPUs.
• Retail and e‑commerce: On‑device product categorization to reduce cloud latency and preserve privacy.
• Robotics: Lightweight perception module for navigation and object identification on resource‑constrained platforms.

Because the model is compatible with CoreML, TensorFlow, and PyTorch, integration can be achieved via native SDKs on iOS, Android, or Linux, allowing developers to embed the model directly into existing pipelines without heavy infrastructure.

Training Details

MobileViT‑S was trained from scratch on the ImageNet‑1k dataset (≈ 1 M images, 1 000 classes). The training pipeline employed a multi‑scale sampler with random image sizes of 160 × 160, 192 × 192, 256 × 256, 288 × 288, and 320 × 320, enabling the model to learn scale‑invariant features.

Preprocessing – Images were randomly resized and cropped, horizontally flipped, and normalized to [0, 1] in BGR order. No complex augmentations (e.g., MixUp or CutMix) were used.

Optimization – Training ran for 300 epochs on 8 NVIDIA GPUs with an effective batch size of 1 024. A learning‑rate warm‑up for the first 3 k steps was followed by cosine annealing. The loss function combined label‑smoothing cross‑entropy with L2 weight decay.

Compute budget – Roughly 2 days on a cluster of 8 × NVIDIA V100 GPUs (≈ 150 TFLOP‑days) were required to reach the reported 78.4 % top‑1 accuracy.

Fine‑tuning – The model can be fine‑tuned on downstream datasets using the same MobileViTForImageClassification class. Because the backbone is lightweight, fine‑tuning typically converges within 20–30 epochs on a single GPU with a batch size of 64.

Related Papers

The primary research foundation for this model is the arXiv pre‑print:

• MobileViT: Light‑weight, General‑purpose, and Mobile‑friendly Vision Transformer – Sachin Mehta & Mohammad Rastegari, 2022.

The paper introduces the MobileViT block, demonstrates its efficiency on ImageNet, and provides the theoretical justification for merging convolutional and transformer operations without positional embeddings. The architecture described in the paper directly maps to the implementation shipped by Apple in the ml‑cvnets repository.

Licensing Information

The repository lists the license as “other” and references the Apple sample code license. This license permits use, modification, and distribution of the code and model weights for research and commercial purposes, provided that the original Apple attribution is retained and that any derivative works include a copy of the license.

Commercial use – The Apple sample code license is permissive enough for commercial products, but you must not claim that Apple endorses your product. You should also avoid redistributing the model under a different license without clear attribution.

Restrictions & requirements

• Include the original copyright notice and a link to the Apple license in any distribution.
• Do not remove or alter the “license: other” tag in the model card.
• For any redistribution of the model files, retain the original file hashes to ensure integrity.