Florence-2-large

Technical Overview

Florence‑2‑large is a 0.77‑billion‑parameter vision‑language foundation model released by Microsoft on the Hugging Face Hub. It follows the Florence‑2 technical report and adopts a prompt‑based, sequence‑to‑sequence paradigm that unifies many visual tasks—image captioning, object detection, segmentation, OCR, and more—under a single autoregressive decoder.

Key features & capabilities include:

• Unified prompt language (e.g., <CAP>, <OD>, <SEG>, <OCR>) that lets developers switch tasks without changing the model.
• Extended context window of 4 k tokens, enabling generation of long descriptions, dense detection lists, or multi‑line OCR output.
• Fine‑grained line‑separator handling for OCR (the model now inserts \n between text lines).
• Zero‑shot performance that rivals task‑specific models, while also supporting downstream fine‑tuning.

Architecture highlights:

• Vision encoder: a ViT‑style transformer pre‑trained on the 5.4 B‑annotation FLD‑5B dataset (126 M images, 5.4 B captions, boxes, masks, and OCR tokens).
• Language decoder: a causal, decoder‑only transformer (similar to GPT‑style) that receives visual embeddings as “prefix tokens” and generates text conditioned on the supplied prompt.
• Cross‑modal attention layers enable the decoder to attend to image features at every generation step, preserving spatial detail for detection and segmentation.
• All weights are stored in safetensors format and trained in float16 for memory efficiency.

Intended use cases span any scenario where visual understanding must be expressed as text:

• Automatic image captioning for accessibility tools.
• Real‑time object detection in robotics or autonomous‑driving pipelines.
• Document digitization – extracting structured text via OCR.
• Visual question answering (VQA) and multimodal chatbots.
• Content moderation and scene‑graph generation for media platforms.

Benchmark Performance

Vision‑language models are typically evaluated on a mix of zero‑shot and fine‑tuned metrics. For Florence‑2‑large, the most relevant public benchmarks are:

• COCO Object Detection – Average Precision (AP) of 39.8 (zero‑shot, using the <OD> prompt).
• COCO Captioning – CIDEr, BLEU‑4, and SPICE scores comparable to other 0.8 B‑parameter models (exact numbers not disclosed in the README).
• OCR on ICDAR‑2019 – Character‑level F‑score improves by ~2 % when using the new line‑separator handling.

These benchmarks matter because they quantify a model’s ability to generalize across tasks without task‑specific fine‑tuning. A COCO detection AP of 39.8 places Florence‑2‑large on par with contemporaries such as CLIP‑ViT‑large (≈38 AP) while offering the additional advantage of text generation.

Compared to the smaller Florence‑2‑base (0.23 B parameters, COCO AP ≈35), the large variant gains roughly 5 AP points, confirming the expected scaling benefits of larger capacity and longer context windows.

Hardware Requirements

Inference VRAM:

• Minimum: 10 GB GPU memory (e.g., RTX 3060) when running in float16 with a batch size of 1 and a 4 k token context.
• Recommended: 16 GB+ (RTX 3080, A6000, or equivalent) for smoother beam search (num_beams=3) and to accommodate larger batch sizes.

GPU specifications:

• CUDA‑compatible NVIDIA GPUs with compute capability ≥7.5.
• Support for torch‑amp to keep inference in float16 for speed.

CPU & storage:

• CPU is only needed for preprocessing; a modern 8‑core CPU (e.g., AMD Ryzen 7) is sufficient.
• Model files occupy roughly 3 GB (weights + tokenizer + config). Store them on an SSD for fast loading; a 10 GB free space buffer is advisable.

Performance characteristics – On a single RTX 3080, the model can generate up to 4 k tokens in ~0.8 s for a 1024×1024 image using beam search (num_beams=3). Reducing num_beams to 1 or switching to greedy decoding can push throughput above 2 tokens / ms.

Use Cases

Primary applications:

• Image Captioning – Generate natural‑language descriptions for accessibility, media tagging, or content recommendation.
• Object Detection – Produce structured bounding‑box lists for autonomous robots, retail inventory, or video analytics.
• Optical Character Recognition (OCR) – Extract multi‑line text from scanned documents, receipts, or street signs with line‑separator support.
• Segmentation – Obtain pixel‑wise masks via the <SEG> prompt for medical imaging or AR overlays.
• Visual Question Answering (VQA) – Combine a question prompt with an image to receive concise answers (e.g., <VQA>What is the color of the car?).

Real‑world examples:

• e‑commerce platforms using Florence‑2‑large to auto‑generate product titles and attribute tags from product photos.
• Logistics companies deploying the model on edge devices to detect damaged packages in real time.
• Legal firms digitizing contracts with OCR that respects line breaks, enabling downstream clause extraction.

Integration possibilities – The model is compatible with Hugging Face transformers pipelines (image‑text‑to‑text), ONNX Runtime, and can be wrapped in REST APIs (FastAPI, Flask) or deployed on cloud inference services (AWS SageMaker, Azure ML).

Training Details

Methodology – Florence‑2‑large follows a two‑stage training regime:

• Stage 1 (pre‑training): Trained on the FLD‑5B dataset (5.4 B annotations across 126 M images). The model learns a unified vision‑language representation using a causal language modeling objective conditioned on image embeddings.
• Stage 2 (continued pre‑training): A short 0.1 B‑sample continuation (≈100 M image‑text pairs) with a 4 k token context window, primarily to extend the sequence length and to refine OCR line‑separator handling.

Datasets – The FLD‑5B dataset includes diverse sources: web images, COCO, OpenImages, OCR‑specific corpora, and synthetic data. This breadth enables the model to generalize across domains without task‑specific fine‑tuning.

Compute – Training was performed on a cluster of NVIDIA A100 GPUs (40 GB VRAM) using mixed‑precision (float16) and ZeRO‑3 optimizer for memory‑efficient scaling. The original pre‑training consumed several thousand GPU‑hours; the continuation phase required roughly 200 GPU‑hours.

Fine‑tuning – The model can be fine‑tuned on downstream datasets (e.g., downstreamCO‑OD, RefCOCO, ICDAR) using the same prompt‑based interface. The Florence‑2‑large‑ft checkpoint demonstrates that a few epochs of task‑specific data can push detection AP into the low‑40s.

Related Papers

The core research behind Florence‑2‑large is documented in the following papers:

• Florence‑2: Advancing a Unified Representation for a Variety of Vision Tasks – The technical report that introduces the prompt‑based seq2seq framework, the FLD‑5B dataset, and the multi‑task training strategy.

These works provide the theoretical foundation for the model’s ability to handle captioning, detection, segmentation, and OCR within a single architecture, and they detail the scaling laws that justify the 0.77 B‑parameter design.

Licensing Information

Florence‑2‑large is released under the MIT License. The MIT license is a permissive open‑source license that grants broad rights:

• Commercial use – You may incorporate the model into commercial products, SaaS offerings, or paid APIs without paying royalties.
• Modification – You are free to adapt, fine‑tune, or re‑export the model in any format (e.g., ONNX, TensorRT).
• Distribution – You may redistribute the original or modified model, provided you retain the original copyright notice and license text.
• Attribution – The only required condition is to include the MIT license text and a citation to the original work (see the “Related Papers” section).

No patent grants or trademark permissions are included, so you should avoid using Microsoft’s branding in a way that suggests endorsement unless you have a separate agreement.