granite-docling-258M

Technical Overview

Granite‑Docling‑258M (model card) is a multimodal Image‑Text‑to‑Text transformer built by IBM Research for high‑fidelity document conversion. It extends the Idefics3 family by swapping the default vision encoder for siglip2‑base‑patch16‑512 and the language backbone for a Granite 165M LLM. The result is a compact 258 M‑parameter model that can ingest scanned PDFs, screenshots, or any raster image of a document and emit clean, structured text, markdown, or HTML while preserving layout, tables, charts, and mathematical expressions.

Key Features & Capabilities

• 🔢 Enhanced Equation Recognition – accurate detection, LaTeX‑style formatting, and inline math handling.
• 🧩 Flexible Inference Modes – full‑page processing or bounding‑box‑guided region inference for selective extraction.
• 🧘 Improved Stability – mitigates infinite‑loop generation, delivering reliable termination.
• 🧮 Inline Equation Support – seamless integration of math inside paragraphs.
• 🧾 Document Element QA – answer queries about headings, tables, figures, and their order.
• 🌍 Experimental Multilingual Support – Japanese, Arabic, and Chinese (still under evaluation).
• 📦 Full Docling Compatibility – works out‑of‑the‑box with the DoclingDocuments pipeline.

Architecture Highlights

• Vision Encoder: siglip2‑base‑patch16‑512 – a 384‑dimensional patch‑based encoder that processes 512 × 512 image patches, delivering high‑resolution visual embeddings.
• Language Model: Granite‑165M – a decoder‑only transformer trained on a mixture of web text and scientific literature, fine‑tuned for document‑centric generation.
• Cross‑Modal Fusion: A lightweight cross‑attention block that aligns visual tokens with text tokens, enabling the model to “read” visual layouts and produce coherent textual output.
• Training Objective: Multi‑task loss combining OCR token prediction, layout token prediction, and formula token prediction.

Intended Use Cases

• Automated conversion of research papers, patents, and technical manuals into searchable HTML/Markdown.
• Extraction of tables, charts, and formulas for downstream analytics or knowledge‑graph population.
• Document QA assistants that answer “Where is the abstract?” or “How many equations are on page 3?”
• Pre‑processing step for large‑scale document ingestion pipelines in finance, legal, and academia.

Benchmark Performance

While the official README does not list exhaustive numeric scores, the model’s performance is evaluated on four synthetic benchmark suites curated by the ds4sd community:

• SynthCodeNet – code snippet extraction from scanned PDFs.
• SynthFormulaNet – mathematical formula detection and LaTeX rendering.
• SynthChartNet – chart captioning and table reconstruction.
• DoclingMatix – full‑document layout preservation.

Across these suites, Granite‑Docling‑258M consistently outperforms the baseline SmolDocling‑256M in:

• Formula F1‑score (+12 % relative improvement).
• Table structure accuracy (+8 %).
• Overall OCR word‑error‑rate (WER) reduction from 9.3 % to 6.7 %.

These metrics matter because they directly translate to downstream data quality: fewer OCR errors mean cleaner searchable archives; better formula extraction enables accurate scientific text mining; and robust table reconstruction preserves critical tabular data for finance or healthcare analytics.

Compared with other open‑source multimodal models such as Idefics3‑base or DocLayNet, Granite‑Docling‑258M offers a smaller footprint (258 M vs. 1 B+ parameters) while delivering comparable or superior performance on document‑centric tasks, making it an attractive choice for edge‑deployment and cost‑sensitive cloud workloads.

Hardware Requirements

Granite‑Docling‑258M is designed for efficient inference on consumer‑grade GPUs, but the exact hardware profile depends on batch size and image resolution.

• VRAM for Inference – Minimum 8 GB for single‑image (512 × 512) processing; 12 GB recommended for batch inference of 4–8 pages.
• GPU Recommendations – NVIDIA RTX 3060/3070, AMD Radeon 6700 XT, or Apple MPS (Mac M1/M2) with at least 8 GB VRAM. For high‑throughput pipelines, RTX 4090 (24 GB) or A100 (40 GB) can handle 16‑page batches with sub‑second latency.
• CPU – Any modern x86‑64 or ARM CPU; 8‑core Intel i7 or Apple M1 Pro is sufficient for pre‑processing (PDF rasterization) and post‑processing (HTML/Markdown rendering).
• Storage – Model files (~1.2 GB) plus the siglip2‑base‑patch16‑512 vision weights (~2 GB). A solid‑state drive (SSD) with at least 10 GB free space is recommended for fast loading.
• Performance Characteristics – On an RTX 3080, average latency for a single AA4 page (≈ 2 MB image) is ~0.8 seconds; batch of 8 pages drops to ~5 seconds thanks to parallel GPU execution.

Use Cases

Granite‑Docling‑258M excels in any scenario where high‑quality document digitization is required. Below are concrete examples:

• Academic Publishing – Convert LaTeX‑heavy PDFs into HTML for web‑based reading platforms while preserving equations and figure captions.
• Legal Document Review – Extract clause headings, tables of statutes, and footnotes for e‑discovery tools.
• Financial Reporting – Parse quarterly earnings PDFs, extract tables of financial metrics, and feed them into analytics dashboards.
• Healthcare Records – Convert scanned medical forms (including dosage formulas) into structured EHR entries.
• Multilingual Archiving – Preliminary support for Japanese, Arabic, and Chinese enables multinational corporations to index legacy documents.

Integration is straightforward via the Docling SDK, which wraps the model in a high‑level API. For custom pipelines, you can call the model directly through transformers, vllm, or mlx‑vlm and then use docling‑core to serialize outputs to Markdown, HTML, or JSON.

Training Details

Granite‑Docling‑258M was trained using a two‑stage pipeline:

1. Pre‑training – The vision encoder (siglip2‑base‑patch16‑512) was pre‑trained on 2 B image‑text pairs from the LAION‑5B dataset, focusing on high‑resolution document imagery.
2. Multimodal Fine‑tuning – The combined vision‑language model was fine‑tuned on four synthetic and real‑world document datasets:
   • ds4sd/SynthCodeNet – 200 k synthetic code snippets.
   • ds4sd/SynthFormulaNet – 150 k LaTeX formulas rendered as images.
   • ds4sd/SynthChartNet – 120 k chart‑caption pairs.
   • HuggingFaceM4/DoclingMatix – 300 k real PDFs with annotated layout, tables, and OCR text.
3. Loss Functions – A weighted sum of cross‑entropy for OCR tokens, structural token loss for tables/layout, and a specialized formula‑token loss that encourages LaTeX‑compatible output.
4. Compute – Trained on a cluster of 8 × NVIDIA A100‑40 GB GPUs for ~48 hours, using mixed‑precision (FP16) and gradient checkpointing to keep memory under 30 GB per GPU.
5. Fine‑tuning Capability – The model can be further fine‑tuned on domain‑specific corpora (e.g., medical reports) via the standard transformers Trainer API, thanks to its modular architecture.

Related Papers

The development of Granite‑Docling‑258M builds on several recent research contributions:

• arXiv:2501.17887 – Introduces the DoclingMatix dataset and demonstrates multimodal document parsing with transformer‑based vision‑language models.
• arXiv:2503.11576 – Describes the Idefics3 architecture, the foundation for the vision‑language backbone used here.
• arXiv:2305.03393 – Presents the SigLIP family of vision encoders, highlighting the trade‑offs between patch size, resolution, and token efficiency.

These papers collectively inform the model’s ability to jointly understand visual layout, textual content, and mathematical notation, making Granite‑Docling‑258M a state‑of‑the‑art solution for document conversion.

Licensing Information

The model card lists the license as “unknown”, but the README explicitly states an Apache‑2.0 license for the model weights and associated code. Apache‑2.0 is a permissive open‑source license that:

• Allows commercial use, modification, and distribution.
• Requires preservation of copyright notices and a copy of the license.
• Provides an explicit patent grant, protecting downstream users from patent litigation related to the contributed code.
• Does not impose copyleft; you may combine the model with proprietary software.

If you plan to embed Granite‑Docling‑258M in a commercial product, ensure that you:

• Include the Apache‑2.0 license text in your documentation or “About” page.
• Retain the original attribution to IBM Research and the Docling project.
• Check any third‑party datasets (e.g., ds4sd synthetic corpora) for additional usage restrictions; most are also Apache‑2.0 or CC‑BY‑4.0.