Technical Overview
Florence‑2‑base is Microsoft’s compact vision‑foundation model that unifies many vision‑language tasks under a single, prompt‑driven sequence‑to‑sequence architecture. Built on a 0.23 Billion‑parameter transformer, the model is pretrained on the massive FLD‑5B dataset—5.4 billion annotations spanning 126 million images—allowing it to learn a rich, multimodal representation that can be queried with simple text prompts such as <CAPTION>, <OD> (object detection), or <SEG> (segmentation).
Key capabilities include:
- Zero‑shot captioning – generate concise or detailed image descriptions without task‑specific fine‑tuning.
- Prompt‑based detection & segmentation – specify the desired output format (boxes, masks, key‑points) directly in the prompt.
- Unified output format – the model always produces a text string that can be post‑processed into structured JSON, making downstream integration straightforward.
Architecture highlights:
- Encoder‑decoder transformer with a vision encoder that processes
pixel_valuesand a causal language decoder that generates token sequences. - All weights are stored in
float16(FP16) to reduce memory while preserving accuracy. - Prompt tokens (e.g.,
<CAPTION>) act as task identifiers, enabling a single model to switch between tasks on the fly.
Intended use cases range from rapid prototyping of image captioning services to building custom object‑detection pipelines that require only a few lines of code. Because the model is relatively small, it can be deployed on a single modern GPU or even on high‑end CPUs for low‑throughput workloads.
Benchmark Performance
Florence‑2‑base is evaluated on a suite of standard vision‑language benchmarks in the original technical report (see arXiv:2311.06242). The most relevant metrics include:
- COCO Caption – CIDEr ≈ 118, BLEU‑4 ≈ 38.5, demonstrating competitive caption quality despite the model’s modest size.
- COCO Object Detection – AP ≈ 35.2 (IoU = 0.5:0.95), showing solid zero‑shot detection performance.
- ADE20K Semantic Segmentation – mIoU ≈ 31.8, confirming the model’s ability to generate accurate mask tokens from prompts.
These benchmarks matter because they reflect real‑world tasks: image description, object localization, and pixel‑level understanding. Compared with larger counterparts (e.g., Florence‑2‑large) and other open‑source vision‑language models such as BLIP‑2, Florence‑2‑base offers a favorable trade‑off between speed, memory footprint, and accuracy, making it a strong candidate for edge‑oriented deployments.
Hardware Requirements
Running Florence‑2‑base efficiently requires a GPU with at least 8 GB of VRAM when using the default FP16 weights. The model’s inference pipeline consumes roughly:
- VRAM: 6–7 GB for the model + 1 GB for the image tensor (224×224 or larger).
- GPU: NVIDIA RTX 3060, RTX 3070, or any GPU with CUDA ≥ 11.0 and support for half‑precision arithmetic.
- CPU: A modern multi‑core CPU (e.g., Intel i7‑9700K or AMD Ryzen 7 3700X) is sufficient for preprocessing; however, CPU‑only inference will be considerably slower.
- Storage: ~1.2 GB for the model checkpoint and tokenizer files; additional space is needed for the FLD‑5B pre‑training data if you plan to fine‑tune.
Typical latency on a single RTX 3070 (FP16) is 30–45 ms per 512×512 image, with batch sizes of 1–4. Scaling to larger batch sizes is linear as long as the GPU memory budget is respected.
Use Cases
Because Florence‑2‑base supports a wide array of prompts, it fits naturally into many real‑world pipelines:
- Content creation tools: Automatic generation of image captions for accessibility or SEO.
- E‑commerce: Zero‑shot product tagging and attribute extraction from catalog images.
- Robotics & automation: On‑the‑fly object detection and segmentation for pick‑and‑place tasks.
- Digital asset management: Bulk annotation of image libraries without manual labeling.
- Education & research: Rapid prototyping of multimodal experiments that require flexible task definitions.
The model’s small footprint also makes it suitable for edge devices (e.g., NVIDIA Jetson) where memory and compute are limited, enabling on‑device inference for privacy‑sensitive applications.
Training Details
Florence‑2‑base was trained on the FLD‑5B dataset—a curated collection of 126 million images annotated with 5.4 billion text‑image pairs covering captions, object boxes, masks, and region descriptions. Training employed a sequence‑to‑sequence objective where the vision encoder processes image patches and the language decoder predicts the target token sequence conditioned on a task‑specific prompt.
- Compute: Approximately 2 M GPU‑hours on Azure NDv4 nodes (8×A100‑40 GB), using mixed‑precision (FP16) to accelerate convergence.
- Optimization: AdamW optimizer with a cosine learning‑rate schedule, warm‑up for the first 10 k steps, and a base learning rate of 1e‑4.
- Fine‑tuning: The
Florence‑2‑base‑ftcheckpoint demonstrates that downstream tasks (e.g., VQA, OCR) can be further refined with as few as 10 k labeled examples, yielding noticeable performance gains.
The model’s architecture and training code are open‑source via the Hugging Face transformers library, enabling researchers to reproduce the results or adapt the training pipeline for domain‑specific data.
Licensing Information
Florence‑2‑base is released under the MIT license. The MIT license is permissive, allowing:
- Free commercial and non‑commercial use.
- Modification, distribution, and sublicensing of the source code and model weights.
- No requirement to disclose source code when deploying the model in a proprietary product.
The only obligation is to retain the original copyright notice and license text in any redistribution. There are no additional usage restrictions (e.g., no‑export‑control clauses) beyond those imposed by the underlying datasets. Consequently, developers can integrate Florence‑2‑base into SaaS platforms, mobile apps, or on‑premise solutions without legal hurdles.