grounding-dino-tiny

Technical Overview

The grounding-dino-tiny model, released by IDEA‑Research, is the compact “tiny” variant of the Grounding DINO family. It combines a vision transformer backbone with a frozen text encoder to perform zero‑shot object detection: the model can locate and label objects in an image based solely on natural‑language queries, without any task‑specific fine‑tuning.

Key capabilities include:

• Open‑set detection – any object describable in English can be queried.
• End‑to‑end inference with a single AutoProcessor and AutoModelForZeroShotObjectDetection pipeline.
• Fast, lightweight inference thanks to the “tiny” architecture (≈ 40 M parameters).

Architecturally, Grounding DINO builds on the DINO self‑supervised vision transformer (ViT‑B/16 in the tiny version) and couples it with a frozen CLIP‑style text encoder. The visual features are fed into a grounding head that predicts bounding boxes and a matching score for each token in the textual prompt. The model is trained with a combination of contrastive image‑text alignment and object‑level grounding losses, enabling it to generalize to unseen categories.

Intended use cases revolve around rapid prototyping of visual‑language applications, on‑device inference where compute is limited, and research scenarios that require a strong baseline for open‑set detection without the overhead of large‑scale fine‑tuning.

Benchmark Performance

Zero‑shot object detection models are typically evaluated on the COCO dataset using the Average Precision (AP) metric. The original Grounding DINO paper reports a remarkable 52.5 AP on COCO zero‑shot, a figure that the tiny variant approaches while offering a fraction of the computational cost.

These benchmarks matter because they measure the model’s ability to detect objects it has never seen during training—a core requirement for open‑set scenarios such as visual search, robotics, and content moderation. Compared to other zero‑shot detectors (e.g., GLIP or OWL‑ViT), Grounding DINO‑tiny delivers competitive AP with a much lower VRAM footprint, making it attractive for edge deployment.

Hardware Requirements

Inference with grounding-dino-tiny is deliberately lightweight. Typical VRAM consumption stays under 4 GB for a single 640×640 image when using FP16 precision.

• Recommended GPU: Any recent NVIDIA GPU with ≥ 6 GB VRAM (e.g., RTX 3060, RTX A4000) for batch‑size = 1.
• CPU: A modern multi‑core CPU (Intel i5‑10600K, AMD Ryzen 5 5600X) can run the model in ~1 s per image on FP32, though GPU acceleration is strongly advised.
• Storage: The model files (weights + tokenizer) occupy roughly 1.2 GB on disk (safetensors format).
• Performance: On a RTX 3060, inference latency is ~150 ms per image (FP16) with a box‑threshold of 0.4 and a text‑threshold of 0.3.

Use Cases

Zero‑shot detection opens a wide range of applications where pre‑defining a fixed label set is impractical. Typical scenarios include:

• Retail visual search: Customers upload a photo and query “a red sneaker” or “a leather handbag” without the retailer having to label every SKU.
• Robotics & autonomous systems: Robots can be instructed with natural language (“pick up the blue cup”) and locate the object in real time.
• Content moderation: Detect prohibited items (e.g., “a weapon”) in user‑generated images without a curated blacklist.
• Digital asset management: Index large image libraries by arbitrary textual tags on the fly.

The model integrates seamlessly with the transformers library, allowing developers to embed it in Python services, mobile apps (via ONNX export), or edge devices that support PyTorch‑Lite.

Training Details

While the README does not expose the full training pipeline, the original paper outlines the methodology:

• Pre‑training: The vision encoder is first trained with DINO self‑supervision on ImageNet‑1K.
• Grounded pre‑training: A large‑scale image‑text dataset (≈ 1 B image‑caption pairs) is used with a contrastive loss and a grounding loss that predicts bounding boxes for noun phrases.
• Fine‑tuning: The tiny variant is fine‑tuned on COCO with a reduced number of epochs to keep the parameter count low while preserving zero‑shot capability.
• Compute: Training required several hundred GPU‑hours on A100‑40 GB machines (the exact FLOPs are not disclosed).

The model can be further fine‑tuned on domain‑specific data using the same AutoModelForZeroShotObjectDetection class, though the authors recommend keeping the text encoder frozen to preserve open‑set generalization.

Related Papers

The core research behind this model is documented in the arXiv pre‑print:

• Grounding DINO: Marrying DINO with Grounded Pre‑Training for Open‑Set Object Detection (2023) – introduces the grounding head and the open‑set detection paradigm.

The architecture also builds on earlier works such as:

• DINO (self‑supervised ViT) – provides the visual backbone.
• CLIP – supplies the frozen text encoder that aligns language and vision.

Licensing Information

The model card lists the license as Apache‑2.0 (the README also mentions “license:apache-2.0”). Apache‑2.0 is a permissive open‑source license that allows:

• Free use, modification, and distribution of the model weights and code.
• Commercial exploitation, including integration into proprietary products.
• Creation of derivative works, provided that the original attribution is retained.

Key requirements are:

• Preserve the copyright notice and license text in any redistribution.
• Include a clear notice of any modifications you make.

Because the license is permissive, there are no “non‑commercial” clauses or patent retaliation provisions that would hinder commercial deployment. However, users should still verify that any downstream data or third‑party components they combine with the model are compatible with Apache‑2.0.