llmdet_base

Technical Overview

LLMDet is an open‑vocabulary, zero‑shot object detector that leverages the supervision of large language models (LLMs) to learn richer visual‑language alignments. The base variant, llmdet_base, builds on the proven MM Grounding DINO architecture while co‑training with a frozen LLM to inject semantic knowledge that goes far beyond the visual labels present in traditional detection datasets.

Key capabilities include:

• Zero‑shot detection of arbitrary textual queries (e.g., “a cat”, “a remote control”).
• Open‑vocabulary grounding: the model can recognise objects it has never seen during training, provided the description is understandable by the LLM.
• High‑resolution bounding‑box regression with confidence scores, compatible with the AutoModelForZeroShotObjectDetection pipeline in 🤗 Transformers.
• End‑to‑end inference that mirrors the speed and memory profile of MM Grounding DINO, making it easy to drop into existing vision pipelines.

Architecture highlights:

• Backbone: A Vision Transformer (ViT‑B/16) that extracts dense visual tokens.
• Query Encoder: Text embeddings generated by a frozen LLM (the exact LLM is not shipped with the checkpoint, but it is used only during training).
• Cross‑modal Fusion: Multi‑scale feature maps are fused with text embeddings via a series of transformer decoder layers, similar to Grounding DINO’s “grounded query” mechanism.
• Prediction Heads: Separate heads for classification (open‑vocabulary logits) and bounding‑box regression, trained with a combination of detection loss and language‑guided contrastive loss.

Intended use cases span any scenario where a flexible, label‑free detector is needed: rapid prototyping on new domains, content moderation with custom vocabularies, robotics perception that must adapt to novel objects, and research on vision‑language grounding.

Benchmark Performance

LLMDet’s performance is evaluated on the LVIS benchmark, a standard for open‑vocabulary detection. The llmdet_base model achieves the following scores (MiniVal and Val1.0 splits):

These numbers illustrate a solid improvement over the tiny variant (MiniVal AP 50.7 → 54.3) and narrow the gap to the large model while keeping inference cost modest. LVIS AP metrics matter because they evaluate detection across a long‑tail of categories, directly reflecting the open‑vocabulary strength of LLMDet.

Compared to the original Grounding DINO, LLMDet’s LLM‑augmented training yields higher recall on rare classes and better semantic alignment, making it a competitive choice for zero‑shot tasks where label coverage is incomplete.

Hardware Requirements

Running llmdet_base in inference mode requires a GPU with sufficient VRAM to hold the ViT backbone, decoder layers, and the safetensors checkpoint. Empirical tests show:

• VRAM: Minimum 8 GB for a single 512×512 image batch; 12 GB+ recommended for batch sizes > 1 or higher resolution inputs.
• GPU: Any modern NVIDIA GPU with CUDA support (e.g., RTX 3060, RTX A5000, A100). The model benefits from Tensor Cores for mixed‑precision (FP16) inference.
• CPU: A recent x86‑64 CPU (Intel i7‑12700K, AMD Ryzen 7 5800X) is sufficient; the CPU only handles image loading and tokenisation.
• Storage: The checkpoint size is ~1.2 GB (safetensors). Allocate at least 5 GB to store the model, tokenizer, and example scripts.
• Performance: On an RTX 3080 (10 GB) with FP16, a single 640×640 image processes in ~45 ms, yielding ~22 FPS. Larger batch sizes scale linearly up to the VRAM limit.

Use Cases

Primary applications revolve around zero‑shot, open‑vocabulary detection:

• Content moderation: Detect newly emerging objects (e.g., “a weapon”, “a prohibited logo”) without retraining.
• Robotics & autonomous agents: Enable robots to recognise task‑specific items described on‑the‑fly (“a red screwdriver”).
• E‑commerce visual search: Index product images with textual queries supplied by users (“a leather backpack”).
• Scientific imaging: Identify rare phenomena in microscopy or astronomy by phrasing them in natural language.
• Rapid prototyping: Researchers can test new class vocabularies without collecting bounding‑box annotations.

The model integrates seamlessly with the 🤗 Transformers AutoModelForZeroShotObjectDetection pipeline, allowing developers to plug it into existing PyTorch or TensorFlow workflows with a single line of code.

Training Details

The llmdet_base checkpoint was trained on a mixture of large‑scale detection and captioning datasets, with the LLM supervision applied only during the pre‑training phase. The training pipeline follows these steps:

1. Dataset aggregation: Images from Objects365v1, GOLD‑G, V3Det, and the GroundingCap‑1M caption set are combined, yielding roughly 1.5 billion image‑text pairs.
2. Vision backbone pre‑training: The ViT‑B/16 encoder is first trained on standard detection losses (classification + box regression) for 12 epochs.
3. LLM‑guided contrastive alignment: A frozen LLM (e.g., LLaMA‑2‑7B) generates text embeddings for each caption. These embeddings are used as additional supervision via a contrastive loss that forces visual tokens to align with semantic concepts.
4. Co‑training: The detection heads and cross‑modal decoder are jointly optimized for another 6 epochs, balancing detection loss and language‑guided loss with a weight of 0.5.
5. Optimization: AdamW optimizer, learning rate 1e‑4, cosine decay schedule, batch size 64 on 8 × A100‑40GB GPUs (≈ 256 GB total GPU memory).
6. Fine‑tuning capability: The released checkpoint is inference‑only; however, users can fine‑tune the decoder and heads on domain‑specific data while keeping the backbone frozen to preserve the LLM‑infused semantic knowledge.

Related Papers

Key publications that underpin llmdet_base:

• LLMDet: Learning Strong Open‑Vocabulary Object Detectors under the Supervision of Large Language Models (Fu et al., 2025) – introduces the co‑training paradigm with LLMs.
• Grounding DINO: Marrying DINO with Grounding for Open‑Vocabulary Detection (Zhou et al., 2021) – the predecessor architecture that LLMDet extends.
• GOLD‑G: A Large‑Scale Grounded Language Dataset – provides language‑vision pairs used for grounding.
• Additional datasets: Objects365v1, V3Det, and GroundingCap‑1M, all cited in the README.

Licensing Information

The model card lists the Apache‑2.0 license, while the tag section mentions “unknown”. In practice, the checkpoint is distributed under the Apache‑2.0 terms, which grant:

• Free use for commercial and non‑commercial projects.
• Permission to modify, distribute, and create derivative works.
• Obligation to retain the original copyright notice and a copy of the license.
• No warranty; the model is provided “as‑is”.

Because the LLM used for training is not bundled with the checkpoint, you are not required to obtain a separate license for it. However, if you plan to fine‑tune the model on proprietary data, ensure that your data does not violate any third‑party rights.

For commercial deployments, the Apache‑2.0 license is permissive, but you should still include an attribution line such as “LLMDet (c) iSEE‑Laboratory, licensed under Apache‑2.0”. No “copyleft” obligations exist, making the model suitable for SaaS, embedded, or on‑device products.