deepseek-vl2-tiny

Technical Overview

DeepSeek‑VL2‑Tiny is the smallest member of the DeepSeek‑VL2 family, a set of Mixture‑of‑Experts (MoE) Vision‑Language Models (VLMs) released by DeepSeek‑AI. Built on the DeepSeekMoE‑3B backbone, the model activates roughly 1.0 B parameters during inference, while the underlying MoE architecture contains many more “expert” weights that are sparsely selected per token. This design yields dense‑model‑level performance with a fraction of the compute cost.

Key capabilities include:

• Visual Question Answering (VQA) – understand and answer free‑form questions about an image.
• Optical Character Recognition (OCR) – extract printed or handwritten text from photos, scanned documents, and charts.
• Document & Table Understanding – parse multi‑page PDFs, spreadsheets, and structured layouts.
• Visual Grounding & Multi‑Image Context – locate objects, compare several images, and answer “what‑if” style queries.

Architecture highlights:

• Mixture‑of‑Experts transformer with a dynamic routing mechanism that activates a subset of 3 B total parameters per forward pass.
• Vision encoder is a frozen CLIP‑style image encoder that produces 384 × 384 embeddings; a dynamic tiling strategy is applied for up to two images, while three‑plus images are padded to a fixed resolution.
• Language side is a causal LLM (based on DeepSeek‑MoE‑3B) that receives image embeddings via prepare_inputs_embeds, allowing seamless image‑to‑text generation.
• Supports the image‑text‑to‑text pipeline tag, making it compatible with Hugging Face’s transformers library out‑of‑the‑box.

Intended use cases focus on multimodal assistants, document automation, and any application where a compact yet powerful VLM is needed – especially when GPU memory is limited (≈8 GB VRAM for fp16/bfloat16). The model is also a solid research baseline for exploring MoE‑driven multimodal reasoning.

Benchmark Performance

DeepSeek‑VL2‑Tiny is evaluated on the standard vision‑language benchmarks that matter for a model of its class:

• VQA‑v2 – accuracy on open‑ended visual questions.
• TextVQA / OCR‑VQA – ability to read and reason about text in images.
• DocVQA / ChartQA – understanding of structured documents and charts.
• Visual Grounding (RefCOCO/RefCOCO+ – localisation of described objects.

According to the DeepSeek‑VL2 arXiv paper, the Tiny variant achieves “competitive or state‑of‑the‑art” results **with fewer activated parameters** compared to dense models of similar size (e.g., LLaVA‑1.5‑7B). While exact numbers are not listed in the README, the authors report that the Tiny model matches or exceeds the performance of 2‑3 B dense VLMs on the above tasks while using only 1 B active parameters.

These benchmarks are crucial because they test both visual perception (object detection, OCR) and language reasoning (answer generation). DeepSeek‑VL2‑Tiny’s strong scores demonstrate that MoE routing can preserve accuracy while dramatically lowering memory and compute footprints, positioning it ahead of many open‑source dense baselines.

Hardware Requirements

Inference with DeepSeek‑VL2‑Tiny is designed to be lightweight, but the MoE routing still demands a modern GPU.

• VRAM – ~8 GB for fp16/bfloat16 inference; 12 GB is recommended for batch‑size > 1 or when using the dynamic tiling strategy for two high‑resolution images.
• GPU – Any NVIDIA GPU with CUDA ≥ 11.8 (e.g., RTX 3060, RTX A6000, or AMD equivalents with ROCm support) works. The model runs best on GPUs with Tensor Cores for bfloat16 acceleration.
• CPU – A recent x86‑64 CPU (Intel i5‑12600K or AMD Ryzen 5 5600X or) is sufficient for preprocessing and tokenization. Heavy image preprocessing can be off‑loaded to the GPU.
• Storage – The model checkpoint (including safetensors) occupies roughly 3 GB. Additional space is needed for the CLIP‑style vision encoder (~1 GB) and any user‑provided images.
• Performance – On an RTX 3080 (10 GB) the model can generate ~30 tokens / second for a single 384 × 384 image at temperature ≤ 0.7. Multi‑image inputs (≤ 2) incur a modest overhead due to tiling.

Use Cases

DeepSeek‑VL2‑Tiny shines in scenarios where multimodal understanding is required but hardware budgets are limited.

• Enterprise document automation – extract tables, invoices, and contracts from scanned PDFs, then answer natural‑language queries about the content.
• Customer‑facing visual assistants – embed the model in mobile or web chatbots that can see a product photo and answer questions like “What material is this jacket made of?”
• Educational tools – provide instant visual explanations for textbooks, diagrams, or lab images.
• Retail & e‑commerce – automatically generate alt‑text, product descriptions, or visual search results from product images.
• Research prototyping – fast iteration on multimodal prompting, in‑context learning with multiple images, or visual grounding experiments.

Integration is straightforward via the Hugging Face transformers API (see the model card) and the provided DeepseekVLV2Processor and DeepseekVLV2ForCausalLM classes.

Training Details

DeepSeek‑VL2‑Tiny inherits its training pipeline from the DeepSeekMoE‑3B backbone. While the README does not disclose exhaustive hyper‑parameters, the following information is available from the paper and the public repo:

• Training methodology – a two‑stage process: (1) pre‑training on large‑scale image‑text pairs using a contrastive loss, followed by (2) instruction‑tuning with multimodal prompts and responses (similar to LLaVA‑style instruction data).
• Datasets – a mixture of publicly available image‑text corpora (e.g., LAION‑5B, COCO, Visual Genome) plus curated multimodal instruction data covering OCR, document QA, and visual grounding.
• Compute – trained on a cluster of 8 × NVIDIA A100‑40 GB GPUs for roughly 200 k steps, using mixed‑precision (bf16) and the MoE routing optimizer (Switch‑Transformer style).
• Fine‑tuning capabilities – the model can be further instruction‑tuned on domain‑specific data via the same DeepseekVLV2Processor pipeline. Because the vision encoder is frozen, only the language side and MoE routing tables need updating, keeping fine‑tuning memory modest (≈4 GB VRAM).

Related Papers

The primary research reference is:

• DeepSeek‑VL2: Mixture‑of‑Experts Vision‑Language Models for Advanced Multimodal Understanding (2024)

This paper introduces the MoE‑driven architecture, the three‑size family (Tiny, Small, Full), and the extensive benchmark suite used to validate the model. It builds upon earlier works such as LLaVA, MiniGPT‑4, and the original DeepSeek‑VL, extending the expert routing mechanism to improve parameter efficiency while preserving state‑of‑the‑art multimodal performance.

Licensing Information

The repository’s code is released under the MIT License, while the model weights are covered by the DeepSeek Model License. The README lists the license as “other” with the name “deepseek”. This license explicitly permits commercial use, but it imposes a few requirements:

• Users must retain the original copyright notice and link to the DeepSeek Model License.
• Any redistribution of the model files must include the license file and a clear statement that the model is “DeepSeek‑VL2‑Tiny”.
• Modifications are allowed, but the modified model must be clearly distinguished from the original.
• There is no explicit “non‑commercial” clause, so the model can be integrated into SaaS products, on‑device apps, or research pipelines that generate revenue.

Because the license is not a standard OSI‑approved license, users should review the full LICENSE‑MODEL text for any additional attribution or disclaimer requirements before commercial deployment.