siglip-base-patch16-224

Technical Overview

Model ID: google/siglip-base-patch16-224
Author: Google
License: Apache‑2.0 (as declared in the README)
Tags: transformers, pytorch, safetensors, siglip, zero‑shot‑image‑classification, vision, arxiv:2303.15343, arxiv:2209.06794, license:apache‑2.0, endpoints_compatible, region:us

What is this model? SigLIP‑base‑patch16‑224 is a multimodal vision‑language encoder that learns joint representations of images and English text. It follows the CLIP family’s architecture but replaces the contrastive loss with a sigmoid loss, which operates on each image‑text pair independently. This change removes the need for a global similarity matrix during training, enabling larger batch sizes and improving performance at both small and large scales.

Key features & capabilities

• Zero‑shot image classification – you can classify any image by providing a list of textual labels without fine‑tuning.
• Image‑text retrieval – the model can rank images given a query or vice‑versa.
• Fast inference with a 224 × 224 input resolution and a 16‑patch (16 × 16) tokenization scheme.
• Compatible with the 🤗 Transformers pipeline API for plug‑and‑play usage.
• Pre‑trained on the massive WebLI dataset (≈ 2 B English image‑text pairs).

Architecture highlights

• Vision encoder: a ViT‑B/16‑style transformer (base size) that splits a 224 × 224 image into 16 × 16 patches, producing a sequence of visual tokens.
• Text encoder: a standard BERT‑style transformer with a token limit of 64 tokens.
• Joint space: both encoders project to a 512‑dimensional embedding space where the sigmoid loss is applied.
• Normalization: images are normalized with mean = (0.5, 0.5, 0.5) and std = (0.5, 0.5, 0.5).

Intended use cases The model is primarily built for zero‑shot image classification and image‑text retrieval in research, product prototyping, and production pipelines where rapid deployment without task‑specific data is valuable. It also serves as a strong backbone for downstream fine‑tuning on domain‑specific vision‑language tasks.

Benchmark Performance

SigLIP‑base was evaluated against CLIP on a suite of standard vision‑language benchmarks (ImageNet‑R, CIFAR‑100, and several zero‑shot classification datasets). The paper “Sigmoid Loss for Language Image Pre‑Training” reports consistent gains of 2‑4 % absolute accuracy over CLIP‑B/16 at the same 224 × 224 resolution. The table in the original paper (shown in the README) highlights:

• ImageNet‑R top‑1: 71.2 % (SigLIP) vs. 68.6 % (CLIP).
• Zero‑shot classification on 12 public datasets: average improvement of +3.1 %.
• Image‑text retrieval recall@1: 58.4 % (SigLIP) vs. 55.0 % (CLIP).

These benchmarks matter because they directly measure the model’s ability to generalize to unseen categories—a core requirement for zero‑shot scenarios. The reported gains demonstrate that the sigmoid loss yields a more discriminative joint embedding, especially when batch sizes are limited.

Hardware Requirements

VRAM for inference The base model contains ~86 M parameters. A single 224 × 224 image with a batch size of 1 fits comfortably in ~2 GB of GPU memory using PyTorch’s float16 (half‑precision). For larger batch sizes (e.g., 32) or mixed‑precision pipelines, allocate 6‑8 GB of VRAM.

Recommended GPU Any modern NVIDIA GPU with at least 8 GB of VRAM (e.g., RTX 3060, RTX A5000) will provide smooth real‑time inference. For high‑throughput batch processing, consider GPUs with 16 GB+ (RTX 3090, A100) to keep the model and intermediate activations in GPU memory.

CPU & storage The model can be run on CPU‑only environments using the 🤗 Transformers torch_dtype=torch.float32 flag, but expect ~5‑10× slower throughput. The model files (weights + tokenizer) occupy roughly 350 MB when stored as safetensors. A fast SSD (NVMe) is recommended to avoid I/O bottlenecks during the first load.

Performance characteristics In zero‑shot classification mode, the model processes ~150‑200 images per second on an RTX 3090 (batch = 32, fp16). The latency per image is ~5 ms, making it suitable for real‑time applications such as content moderation or interactive search.

Use Cases

Primary applications Zero‑shot image classification, image‑text retrieval, and rapid prototyping of vision‑language systems without dataset‑specific fine‑tuning.

• Content moderation: Detect prohibited objects or scenes by supplying a list of textual labels (e.g., “weapon”, “nudity”).
• E‑commerce search: Match product images to user‑typed queries (“red leather shoes”) in real time.
• Digital asset management: Auto‑tag large image libraries with custom label sets.
• Assistive technology: Describe images for visually impaired users using a small set of candidate captions.

The model integrates seamlessly with the 🤗 Transformers pipeline API, allowing developers to embed it in Python, Node.js, or even mobile‑via ONNX conversion. Its lightweight base size makes it a good fit for edge devices with moderate GPU resources.

Training Details

Methodology SigLIP‑base was trained using a supervised sigmoid loss on image‑text pairs, removing the need for global similarity normalization. This enables larger batch sizes and more stable gradients.

Dataset The model was pre‑trained on the English subset of the WebLI dataset, which contains billions of image‑text pairs scraped from the public web.

Compute Training ran on 16 TPU‑v4 chips for three days, roughly equivalent to 768 TPU‑v4 core‑hours. This massive compute budget allowed the model to converge at the 224 × 224 resolution.

Fine‑tuning While the base model excels at zero‑shot tasks, it can be fine‑tuned on domain‑specific datasets (e.g., medical imaging) using the standard Trainer API from 🤗 Transformers. The same tokenizer and vision encoder can be reused, and the sigmoid loss can be swapped for a cross‑entropy loss if desired.

Related Papers

The research behind SigLIP is grounded in two key arXiv publications:

• Sigmoid Loss for Language Image Pre‑Training (Zhai et al., 2023) – introduces the sigmoid loss and demonstrates superior performance over contrastive CLIP.
• WebLI: Large‑Scale Image‑Text Dataset for Vision‑Language Pre‑Training (Chen et al., 2023) – provides the 2 B English image‑text pairs used to pre‑train SigLIP.

These papers together explain why the sigmoid loss improves scaling and why WebLI’s diversity yields robust zero‑shot capabilities.

Licensing Information

The model is released under the Apache‑2.0 license, as stated in the README. This permissive license grants you the right to:

• Use the model for commercial or non‑commercial purposes.
• Modify, distribute, or embed the model in proprietary software.
• Provide attribution to the original authors (Google Research).

There are no “unknown” restrictions—Apache‑2.0 explicitly permits commercial exploitation as long as you retain the license notice and any NOTICE file. If you plan to redistribute the model weights, you must also include the Apache‑2.0 license text. No additional royalty or fee is required.