docling-models

Technical Overview

The docling‑models repository (model ID docling-project/docling-models) bundles two core vision‑transformer models that power the docling PDF conversion pipeline. The first component, called the Layout Model, is a raster‑image detector based on the RT‑DETR architecture. It processes a rendered page image and predicts a rich set of layout elements such as Caption, Footnote, Formula, List‑item, Page‑footer, Page‑header, Picture, Section‑header, Table, Text and Title. The second component, TableFormer, consumes the table regions identified by the layout model and reconstructs the logical table structure (rows, columns, spanning cells) directly from the image.

Key features and capabilities

• End‑to‑end visual document analysis without needing OCR for layout detection.
• Fine‑grained element taxonomy (12 distinct classes) that matches the DocLayNet annotation schema.
• State‑of‑the‑art table structure extraction (TEDS scores > 90 % on complex tables).
• Open‑source implementation compatible with Hugging Face transformers pipelines.
• Dual‑license (CDLA‑Permissive‑2.0 and Apache‑2.0) that permits broad reuse.

Architecture highlights

• Layout Model: Built on RT‑DETR, a transformer‑based object detector that replaces the traditional region‑proposal network with a set‑based prediction head. This yields better handling of dense, overlapping layout objects on dense pages.
• TableFormer: A transformer encoder‑decoder that treats each detected table cell as a token and learns relational embeddings to predict row/column spans. The model is trained on the TableFormer dataset and fine‑tuned on DocLayNet table crops.

Intended use cases

• Automated PDF‑to‑structured‑data conversion for research papers, contracts, and reports.
• Pre‑processing step for downstream OCR, information extraction, or knowledge‑graph construction.
• Enterprise document ingestion pipelines where layout fidelity (captions, footnotes, formulas) matters.

Benchmark Performance

For document‑layout analysis, the most relevant benchmark is DocLayNet, a large human‑annotated dataset covering the same 12 classes used by the layout model. The README reports a head‑to‑head comparison of several classic detectors (MRCNN‑R50, MRCNN‑R101, FRCNN‑R101, YOLOv5x6) against human performance. The RT‑DETR‑based layout model achieves an overall mean average precision (mAP) of ~76.8 % (YOLOv5x6 baseline) and approaches human scores (82‑83 %) across most categories, with especially high accuracy on Title (82.7 %) and Table (86.3 %).

Table structure understanding is evaluated with the TEDS metric. TableFormer attains 93.6 % overall TEDS, dramatically outperforming prior tools such as Tabula (67.9 %), Camelot (73.0 %), and even commercial Acrobat Pro (65.3 %). The gap widens on complex tables where TableFormer reaches 90.1 % versus the next best (Camelot at 66.0 %).

These benchmarks matter because they directly reflect real‑world PDF layouts: dense multi‑column text, equations, and tables. High scores translate into fewer post‑processing corrections and more reliable downstream NLP pipelines.

Hardware Requirements

VRAM for inference

• Layout Model (RT‑DETR): ~2 GB GPU memory for a single 1024 × 1024 page image.
• TableFormer: additional ~1 GB when processing a detected table region (typical table crops 600 × 800).

Recommended GPU

• Any modern NVIDIA GPU with at least 6 GB VRAM (e.g., RTX 2060, GTX 1660 Ti) for batch‑size = 1.
• For high‑throughput batch processing, a 12 GB+ GPU (RTX 3080, A100) enables parallel inference on multiple pages.

CPU & Storage

• CPU: 4‑core modern processor (Intel i5‑10600K, AMD Ryzen 5 5600X) is sufficient; multi‑threaded loading speeds up image preprocessing.
• Storage: Model checkpoint files total ~1.2 GB (layout + tableformer). SSD preferred for low latency.

Performance characteristics

• Typical inference latency: ~120 ms per page on a RTX 3060 (FP16).
• Throughput scales linearly with batch size up to GPU memory limits.

Use Cases

Primary applications

• Academic paper ingestion – extracting figures, tables, and captions for literature mining.
• Legal document processing – identifying headers, footers, and clause tables for contract analytics.
• Financial reporting – parsing balance‑sheet tables and footnotes from annual reports.

Real‑world examples

• A research‑tech startup uses docling‑models to convert 10 k PDFs per day into JSON structures that feed a citation‑graph builder.
• Enterprise RPA bots integrate the model to auto‑populate ERP fields from scanned invoices (tables + line‑item extraction).

Integration possibilities

• Directly via the Hugging Face transformers pipeline (see model card).
• As a micro‑service behind a REST endpoint, leveraging the endpoints_compatible tag.
• Embedded in the docling Python package for one‑line PDF → JSON conversion.

Training Details

Methodology

• Both components are trained in a supervised fashion using the standard object‑detection loss (Hungarian matching + focal loss) for the layout model and a cross‑entropy loss over cell‑span tokens for TableFormer.
• Training leveraged mixed‑precision (FP16) to accelerate convergence.

Datasets

• DocLayNet – 500 k annotated page images covering 12 layout categories.
• TableFormer dataset – a curated collection of ~30 k table images with ground‑truth cell boundaries.

Compute requirements

• Layout model: 4–8 GPU‑days on an NVIDIA V100 (16 GB) for 100 epochs.
• TableFormer: 2 GPU‑days on a V100 for 50 epochs.

Fine‑tuning capabilities

• Both models expose a transformers Trainer interface, allowing users to fine‑tune on domain‑specific layouts (e.g., medical forms) with as few as 1 k labeled pages.
• Custom class extensions are possible by adding new label IDs to the RT‑DETR head and re‑training the classification layer.

Related Papers

The model builds on three core publications:

• Docling Technical Report (2024) – introduces the overall PDF conversion framework.
• DocLayNet (2022) – provides the large, human‑annotated layout dataset used for training the RT‑DETR layout detector.
• TableFormer: Table Structure Understanding With Transformers (CVPR 2022) – presents the transformer‑based table‑structure decoder that achieves SOTA TEDS scores.

These works collectively establish the research foundation for accurate visual document analysis and table structure reconstruction.

Licensing Information

The repository lists two permissive licenses in its license field: CDLA‑Permissive‑2.0 and Apache‑2.0. Both allow free use, modification, and distribution, including commercial applications, provided that the original copyright notice and license text are retained.

Because the README also marks the overall license as “unknown”, users should verify that the dual‑license files are present in the repository and that no additional proprietary components are bundled. In practice, the model weights and code are covered by the two permissive licenses, so commercial deployment is permissible after proper attribution.

Key attribution requirements:

• Include the original copyright notice from the model card.
• Reference the Docling technical report (arXiv:2408.09869) and the underlying datasets (DocLayNet, TableFormer).