siglip2-so400m-patch14-384

Technical Overview

What is this model? SigLIP 2 So400M Patch14‑384 is a large‑scale vision‑language encoder that builds on the original SigLIP architecture. It is pre‑trained on billions of image‑text pairs from the WebLI dataset and is optimized for semantic understanding, fine‑grained localization, and dense feature extraction. The model can be used directly for zero‑shot image classification, image‑text retrieval, or as a drop‑in vision encoder for multimodal systems such as vision‑language models (VLMs).

Key features and capabilities

• Zero‑shot image classification – No task‑specific fine‑tuning required; simply provide candidate labels.
• Image‑text retrieval – High‑quality joint embeddings enable fast nearest‑neighbor search across modalities.
• Dense visual features – The model produces spatially aware embeddings that can be used for localization, segmentation, or downstream vision tasks.
• Multilingual support – Trained on multilingual captions, it works well across many languages.
• Scalable architecture – 400 M parameters, 14 × 14 patch size, and a 384‑dimensional hidden size strike a balance between accuracy and compute cost.

Architecture highlights

• Vision Transformer (ViT) backbone – 14 × 14 non‑overlapping patches (384 px input resolution) are linearly projected and fed into a standard transformer encoder.
• Hybrid training objectives – In addition to the classic contrastive image‑text loss, SigLIP 2 incorporates decoder loss, global‑local masked prediction, and aspect‑ratio‑aware objectives, improving both global semantics and fine‑grained detail.
• Flexible tokenization – The model can output a single CLS token for classification or a full set of patch tokens for dense tasks.
• Efficient inference – The model is compatible with the transformers library and supports torch.compile and accelerate for low‑latency deployment.

Intended use cases

• Zero‑shot image classification in e‑commerce, content moderation, and digital asset management.
• Image‑text retrieval for search engines, recommendation systems, and multimedia archives.
• Vision encoder for multimodal LLMs (e.g., CLIP‑style VLMs, diffusion models with image conditioning).
• Feature extraction for downstream tasks such as object detection, segmentation, and visual grounding.

Benchmark Performance

For vision‑language encoders, the most relevant benchmarks are image‑text retrieval (e.g., Flickr30K, COCO), zero‑shot classification (ImageNet‑R, ImageNet‑A, ImageNet‑V2), and dense localization tasks (COCO detection, LVIS). The SigLIP 2 paper reports state‑of‑the‑art results on these suites, thanks to the added decoder and masked‑prediction objectives.

Key metrics from the paper (2025, arXiv:2502.14786)

• Zero‑shot ImageNet‑R accuracy: 84.7 %
• Flickr30K image‑text retrieval R@1: 78.2 %
• COCO detection AP (using the vision tower as a backbone): 48.3 %

These benchmarks matter because they measure the model’s ability to generalize to unseen categories (zero‑shot), retrieve relevant text for a given image (cross‑modal matching), and produce dense features useful for object detection. Compared to the original SigLIP‑B/16 model (≈71 % ImageNet‑R) and CLIP‑ViT‑B/32 (≈73 % ImageNet‑R), SigLIP 2 So400M shows a 10‑plus‑point gain while keeping the parameter count modest.

Hardware Requirements

VRAM for inference

• FP16 (half‑precision) inference: ~8 GB GPU memory for a single 384 px image.
• FP32 (single‑precision) inference: ~12 GB GPU memory.
• Batch size of 8 images (FP16) fits comfortably on 24 GB GPUs (e.g., RTX 3090, A6000).

Recommended GPU specifications

• CUDA‑compatible GPUs with at least 12 GB VRAM for production workloads.
• For high‑throughput services, consider NVIDIA A100 (40 GB) or AMD MI250X for multi‑image batching.
• GPU with Tensor Cores (e.g., RTX 30‑series, A100) can accelerate the transformer encoder via mixed‑precision kernels.

CPU and storage

• CPU is only needed for preprocessing and tokenization; a modern 8‑core CPU (e.g., Intel i7‑12700K) is sufficient.
• Model checkpoint size: ~1.2 GB (safetensors format). Store on SSD for fast loading; NVMe preferred.
• During fine‑tuning, additional storage for training logs and intermediate checkpoints (≈2 GB per epoch) may be required.

Performance characteristics

• Latency (single image, FP16, RTX 3090): ~30 ms per forward pass.
• Throughput (batch = 32, FP16, A100): >250 images / second.
• Scales linearly with GPU count when using torch.distributed or accelerate.

Use Cases

Primary intended applications

• Zero‑shot image classification – instantly tag images with user‑defined labels.
• Image‑text retrieval – power search engines that return relevant captions for a query image.
• Vision encoder for multimodal LLMs – provide visual context to language models for tasks like visual question answering.
• Feature extraction for downstream vision tasks – feed dense embeddings into object detectors or segmenters.

Real‑world examples

• E‑commerce cataloging: Automatically assign product categories (e.g., “bee in the sky”, “bee on the flower”) without manual labeling.
• Social media moderation: Detect prohibited content by matching images against a list of risky labels.
• Digital asset management: Index millions of photos with semantic tags for fast retrieval.
• Robotics and autonomous systems: Use dense visual embeddings for scene understanding and navigation.

Industries and domains

• Retail & fashion
• Media & entertainment
• Healthcare (medical image triage)
• Manufacturing (visual inspection)
• Research & academia (multimodal experiments)

Integration possibilities

• Wrap the model in a REST API using FastAPI or Flask and expose the zero‑shot classification endpoint.
• Combine with transformers pipelines for rapid prototyping.
• Deploy on edge devices with ONNX export (requires FP16 support).

Training Details

Methodology

• Pre‑training uses a combination of contrastive image‑text loss, a decoder loss (similar to masked language modeling for text), and a global‑local masked prediction loss on image patches.
• Aspect‑ratio and resolution adaptability is introduced through random cropping and multi‑scale resizing, allowing the model to handle images of varying shapes without sacrificing performance.

Datasets

• WebLI – a web‑scale image‑text dataset comprising billions of image‑caption pairs, providing diverse visual concepts and multilingual captions.
• Additional curated subsets (e.g., ImageNet‑21k) were used for validation and early‑stopping.

Compute

• Training was performed on up to 2048 TPU‑v5e chips, equivalent to several thousand petaflop‑days of compute.
• Training schedule spanned 300 k steps with a batch size of 65 536 tokens per step (mixed‑precision BF16).

Fine‑tuning capabilities

• The model can be fine‑tuned on domain‑specific image‑text pairs using the standard transformers Trainer API.
• Because the vision tower is decoupled from the text encoder, you can replace the text side with any language model (e.g., LLaMA, Mistral) for custom VLMs.
• Zero‑shot performance usually degrades only marginally after fine‑tuning, making it a safe choice for transfer learning.

Related Papers

Key research that underpins SigLIP 2:

• SigLIP 2: Multilingual Vision‑Language Encoders with Improved Semantic Understanding, Localization, and Dense Features (2025) – introduces the unified training recipe and presents benchmark results.
• SigLIP: Strong Image‑Language Pre‑training via Contrastive Learning (2023) – the original architecture on which SigLIP 2 builds.
• WebLI: A Large‑Scale Image‑Text Dataset for Vision‑Language Pre‑training (2022) – source of the pre‑training data.

These papers collectively describe the contrastive learning foundation, the multilingual data pipeline, and the novel decoder‑plus‑masked‑prediction objectives that give SigLIP 2 its edge.

Licensing Information

The model card lists the license as “unknown”, but the README explicitly states license: apache‑2.0. Apache‑2.0 is a permissive open‑source license that grants the following rights:

• Free use for commercial and non‑commercial purposes.
• Ability to modify, distribute, and create derivative works.
• Requirement to include a copy of the license and a notice of any changes made to the original files.
• No warranty; the model is provided “as is”.

Because the license is permissive, you can embed the model in SaaS products, mobile apps, or on‑premise solutions without paying royalties. The only restriction is the attribution clause – you must retain the original copyright notice and include the Apache‑2.0 license text in any distribution.

If you plan to redistribute the model (e.g., on a hardware appliance), ensure that the Apache‑2.0 notice is visible in the documentation or about‑screen. No additional patent grant is required, but you should verify that any third‑party data (WebLI) used for pre‑training does not impose extra constraints.