siglip-so400m-patch14-384

Technical Overview

What is this model? google/siglip-so400m-patch14-384 is a vision‑language transformer that follows the CLIP paradigm – it learns joint embeddings for images and text. It is built on the SoViT‑400M architecture, a shape‑optimized variant of the Vision Transformer (ViT) that was introduced to maximize compute‑efficiency at a fixed parameter budget. The model is pre‑trained on the large‑scale WebLI dataset at a native resolution of 384 × 384 pixels and uses a sigmoid‑based contrastive loss instead of the traditional softmax‑cross‑entropy loss used by CLIP.

Key features and capabilities

• Zero‑shot image classification – no task‑specific fine‑tuning required.
• Image‑text retrieval – can rank images given a textual query and vice‑versa.
• High‑resolution input (384 px) with a balanced trade‑off between accuracy and compute.
• Sigmoid loss enables larger batch sizes and better performance at small batch sizes.
• Compatible with Hugging Face pipeline API and AutoProcessor/AutoModel classes.

Architecture highlights

• SoViT‑400M – a 400 M‑parameter Vision Transformer whose depth, width, and patch size (14 × 14) are tuned for optimal compute‑to‑accuracy scaling.
• Patch embedding – images are split into 14 × 14 patches, projected, and fed to the transformer encoder.
• Text encoder – a standard transformer‑based language encoder that tokenizes up to 64 tokens per caption.
• Sigmoid contrastive loss – operates on each image‑text pair independently, removing the need for global similarity normalization.

Intended use cases

• Zero‑shot classification of arbitrary image collections.
• Content‑based image search in e‑commerce or digital asset management.
• Multimodal retrieval for research and prototyping.
• Rapid prototyping of vision‑language applications without labeled data.

Benchmark Performance

For vision‑language models, the most informative benchmarks are zero‑shot image classification accuracy on datasets such as ImageNet, CIFAR‑10/100, and the WebLI retrieval tasks. The original SigLIP paper reports a consistent 2‑3 % gain over CLIP‑ViT‑B/32 on ImageNet‑1K zero‑shot accuracy, and similar improvements on downstream retrieval metrics (e.g., Recall@1, Recall@5).

The README includes a comparative table (see the image in the model card) that shows SigLIP‑SoViT‑400M surpasses the baseline CLIP model of comparable size across a suite of vision‑language benchmarks, confirming the effectiveness of the sigmoid loss and the shape‑optimized architecture.

These benchmarks matter because they directly reflect how well the model can generalize to unseen categories—a core requirement for zero‑shot and retrieval scenarios. Compared to other open‑source CLIP‑style models (e.g., openai/clip-vit-base-patch32), siglip‑so400m‑patch14‑384 delivers higher accuracy while keeping the parameter count modest, making it a strong candidate for production‑grade deployments.

Hardware Requirements

• VRAM for inference – The model’s checkpoint (≈ 2 GB in safetensors format) plus the 384 × 384 image tensor typically requires 8 GB of GPU memory for a batch size of 1. Batch sizes of 8‑16 can be accommodated on 16 GB GPUs.
• Recommended GPU – NVIDIA RTX 3080/3090, RTX A6000, or any recent AMD GPU with ≥ 12 GB VRAM. For large‑scale batch inference, consider GPUs with 24 GB+ (e.g., RTX 4090, A100).
• CPU requirements – No special CPU features are needed; a modern multi‑core processor (≥ 8 cores) is sufficient to feed the GPU when using the AutoProcessor for image decoding and tokenization.
• Storage – The model files (weights, tokenizer, config) occupy roughly 2 GB on disk. An additional ~500 MB is needed for the example image and temporary tensors.
• Performance characteristics – On a single RTX 3080, inference latency for a single 384 × 384 image is ~ 30 ms (batch = 1). Throughput scales linearly with batch size until VRAM limits are reached.

Use Cases

Zero‑shot image classification is the primary intended application, but the model’s multimodal embeddings enable a broader set of scenarios:

• Content moderation – Classify images for policy‑violating content without curating a labeled dataset.
• E‑commerce search – Match user‑typed queries (“red leather shoes”) to product images in real time.
• Digital asset management – Tag large image libraries automatically for better organization.
• Research prototyping – Quickly evaluate vision‑language hypotheses without expensive annotation pipelines.
• Cross‑modal retrieval – Build “search by image” or “search by text” features for media platforms.

Training Details

Methodology – The model was trained on image‑text pairs using the sigmoid loss, which treats each pair independently and eliminates the need for global similarity normalization. This enables larger batch sizes and more stable training dynamics.

Dataset – Pre‑training leveraged the WebLI dataset, a curated collection of billions of image‑text pairs harvested from the web, covering a wide variety of visual concepts and linguistic descriptions.

Compute – Training was performed on 16 TPU‑v4 chips for three days, reflecting a substantial compute budget that allowed the model to converge at the 384 × 384 resolution.

Fine‑tuning – While the model is released primarily for zero‑shot use, it can be fine‑tuned on downstream tasks (e.g., domain‑specific classification) using the standard Trainer API in transformers. The same tokenizer and processor can be reused, and the model’s weights are compatible with mixed‑precision training.

Related Papers

The model builds on three core research contributions:

• Sigmoid Loss for Language Image Pre‑Training (Zhai et al., 2023) – Introduces the sigmoid contrastive loss that underpins SigLIP’s training.
• Getting ViT in Shape: Scaling Laws for Compute‑Optimal Model Design (Alabdulmohsin et al., 2023) – Describes the SoViT‑400M shape‑optimized architecture.
• WebLI: A Large‑Scale Image‑Text Dataset for Language‑Image Pre‑Training (Chen et al., 2023) – Provides the massive WebLI dataset used for pre‑training.

Licensing Information

The model card lists the license as apache‑2.0 in the tags, while the top‑level license field is marked “unknown”. In practice, the Apache 2.0 license applies to the underlying code and weights, granting the following permissions:

• Free use, modification, and distribution, both personal and commercial.
• No royalty or fee requirements.
• Permission to create derivative works, provided that a copy of the Apache 2.0 license is included.

Because the license is permissive, you may integrate the model into commercial products, SaaS platforms, or embedded systems. The only mandatory condition is attribution: you must retain the copyright notice and include a copy of the Apache 2.0 license in any distribution of the model or derived works.