deit-tiny-patch16-224

Technical Overview

The facebook/deit-tiny-patch16-224 model is a compact Vision Transformer (ViT) variant known as DeiT‑tiny. It has been pre‑trained and fine‑tuned on the ImageNet‑1k dataset (≈1 M images, 1 000 classes) at a canonical resolution of 224 × 224 pixels. The model accepts an image, splits it into 16 × 16 pixel patches, linearly embeds each patch, and feeds the resulting token sequence through a stack of transformer encoder layers. A dedicated [CLS] token aggregates global information and is used for classification.

Key features and capabilities

• Very small footprint – only ~5 M trainable parameters.
• Data‑efficient training: achieves 72.2 % top‑1 ImageNet accuracy with far fewer FLOPs than classic ViT‑base.
• Fully compatible with the Hugging Face ViTModel and ViTForImageClassification pipelines.
• Supported feature extractor: DeiTFeatureExtractor (or AutoFeatureExtractor).
• Ready‑to‑use for both PyTorch and, soon, TensorFlow/JAX back‑ends.

Architecture highlights

• Patch size: 16 × 16 pixels → 196 patches + 1 [CLS] token.
• Embedding dimension: 192.
• Depth: 12 transformer encoder layers.
• Number of attention heads: 3.
• MLP hidden size: 768.
• LayerNorm and GELU activation throughout.

Intended use cases

• Fast image‑level classification on edge devices or low‑power GPUs.
• Feature extraction for downstream vision tasks (e.g., object detection, segmentation) via the [CLS] embedding.
• Baseline model for research on data‑efficient training or knowledge distillation.

Benchmark Performance

For vision transformers, the most informative benchmarks are ImageNet top‑1 and top‑5 accuracy, model size (parameter count), and inference latency. The DeiT‑tiny variant reports the following results on the ImageNet‑1k validation set:

• Top‑1 accuracy: 72.2 %
• Top‑5 accuracy: 91.1 %
• Parameter count: 5 M

These numbers are impressive for a model of this size, outperforming the original ViT‑tiny (≈66 % top‑1) while using a comparable amount of compute. Compared with the larger DeiT‑small (22 M params, 79.9 % top‑1) and DeiT‑base (86 M params, 81.8 % top‑1), DeiT‑tiny offers a sweet spot for latency‑critical applications where memory and power budgets are tight.

Hardware Requirements

VRAM for inference

• Approximately 1 GB of GPU memory is sufficient for a single 224 × 224 image batch (batch‑size = 1).
• Batch sizes up to 32 fit comfortably on a 8 GB GPU (e.g., RTX 2070, GTX 1080 Ti).

Recommended GPU specifications

• CUDA‑compatible GPU with at least 4 GB VRAM for production workloads.
• For high‑throughput serving, a GPU with 8 GB+ (e.g., RTX 3060, A100) enables larger batch processing.

CPU requirements

• Modern x86‑64 CPUs (Intel i5‑10600K, AMD Ryzen 5 5600X) can run inference at ~30 ms per image when using the PyTorch CPU backend.
• AVX‑512 or NEON acceleration (on ARM) further reduces latency.

Storage needs

• Model checkpoint size: ~400 MB (including configuration and tokenizer files).
• Additional space for the feature extractor (~2 MB) and optional tokenizer files.

Performance characteristics

• Inference latency: ~10 ms per image on a single RTX 3080 (FP16).
• Throughput: >100 images / second on a 4‑GPU server (batch‑size = 64).

Use Cases

Primary intended applications

• Real‑time image classification on mobile or embedded devices.
• Feature extraction for downstream vision pipelines (e.g., object detection, image retrieval).
• Baseline for research on model compression, quantization, and knowledge distillation.

Real‑world examples

• Smart‑camera systems that need to identify objects within a few milliseconds.
• Retail checkout kiosks that classify products from a limited camera view.
• Wildlife monitoring stations that run on solar‑powered edge hardware.

Industries or domains

• Retail & e‑commerce (product tagging, visual search).
• Manufacturing (defect detection on assembly lines).
• Healthcare (quick triage of medical images when high accuracy is not critical).

Integration possibilities

• Plug‑and‑play with Hugging Face pipeline("image-classification") for rapid prototyping.
• Export to ONNX or TorchScript for deployment in C++/Rust inference servers.
• Fine‑tune on domain‑specific datasets using the same ViTForImageClassification class.

Training Details

Training methodology

• Supervised pre‑training on ImageNet‑1k with a standard cross‑entropy loss.
• Knowledge distillation from a stronger teacher (a ResNet‑152) using a distillation token, as described in the DeiT paper.
• Optimized with AdamW, cosine learning‑rate schedule, and weight decay of 0.05.

Datasets used

• ImageNet‑1k (≈1 M training images, 1 000 classes) – the only dataset used for both pre‑training and fine‑tuning.

Compute requirements

• Training performed on a single 8‑GPU node (NVIDIA V100 or equivalent) for roughly 3 days.
• Batch size of 1024 (distributed across 8 GPUs) and a learning rate of 0.001.

Fine‑tuning capabilities

• The model can be fine‑tuned on any image classification dataset by replacing the final classification head.
• Because the backbone is small, fine‑tuning converges quickly (often < 10 epochs on a 10 K‑image dataset).

Related Papers

The model builds on two seminal works:

• Training data‑efficient image transformers & distillation through attention (Touvron et al., 2020) – introduces the DeiT training paradigm and the distillation token.
• An image is worth 16×16 words: Transformers for image recognition at scale (Dosovitskiy et al., 2020) – the original Vision Transformer (ViT) architecture.

These papers demonstrate that Transformers can match or exceed CNN performance when trained efficiently, and they provide the theoretical and experimental foundation for the tiny‑sized DeiT variant.

Licensing Information

The model is distributed under the Apache 2.0 license, as indicated in the README. Apache 2.0 is a permissive open‑source license that grants users the right to use, modify, distribute, and sell the software, provided that the following conditions are met:

• Preserve the original copyright notice and license text in any redistributed copies.
• Provide a clear notice of any modifications you make.
• Include a copy of the license in binary distributions.

Because the license is permissive, commercial use is allowed without paying royalties. The only restriction is the requirement for attribution and the inclusion of the license file. No patent grant is explicitly stated, but Apache 2.0 includes an implicit patent‑license clause that protects downstream users from patent claims by contributors.