docling-layout-heron

Technical Overview

docling-layout-heron is the default layout‑analysis engine of the Docling ecosystem. It is a high‑resolution object‑detection model that parses scanned or digital pages and returns a structured description of every visual element – captions, tables, figures, footnotes, headers, and even interactive widgets such as checkboxes. In practice, the model turns a raster image of a document into a machine‑readable layout graph that downstream pipelines can use for OCR, information extraction, or document conversion.

Key features and capabilities include:

• Detects 17 distinct layout classes (e.g., Caption, Formula, Key‑Value Region).
• Supports both printed and handwritten content thanks to a diverse training corpus.
• Provides pixel‑accurate bounding boxes with confidence scores, enabling fine‑grained post‑processing.
• Optimized for the rt_detr_v2 family, offering a good trade‑off between speed and accuracy.
• Fully compatible with the transformers and torch pipelines, requiring only a few lines of code.

Architecture highlights:

• The backbone is a RT‑DETR‑v2 transformer that replaces the classic CNN encoder with a lightweight, recurrent‑token transformer, reducing memory footprint while preserving spatial resolution.
• Object queries are learned per class, allowing the model to specialize on the 17 layout categories without needing a separate detection head per class.
• Training employs a combination of Hungarian matching loss and focal loss, a strategy proven to improve detection of small or densely packed elements such as footnotes or list items.

Intended use cases:

• Pre‑processing for OCR engines (e.g., Tesseract, EasyOCR) to isolate text blocks.
• Automatic document conversion to HTML, LaTeX, or structured JSON.
• Enterprise document‑management systems that need to index and search by layout semantics (e.g., “find all tables in annual reports”).
• Academic publishing pipelines that must extract formulas, figures, and captions for metadata generation.

Benchmark Performance

For layout‑analysis models, the most relevant benchmarks are mean Average Precision (mAP) at IoU thresholds of 0.5 and 0.75, as well as class‑wise recall on heterogeneous document corpora. The authors of Advanced Layout Analysis Models for Docling (arXiv:2509.11720) report an overall mAP@0.5 of **0.78** and a mAP@0.75 of **0.62** on a held‑out test set that mixes scientific papers, invoices, and legal contracts. The model also achieves > 90 % recall on the “Text” and “Title” classes, while maintaining > 80 % precision on smaller objects such as “Checkbox‑Selected” and “Formula”.

These metrics matter because they directly translate to downstream accuracy: a missed caption can break a citation chain, and a false‑positive table can corrupt data extraction pipelines. Compared with earlier Docling models (e.g., the docling-layout-kraken baseline), heron improves mAP@0.5 by roughly **12 %**, while keeping inference latency under 150 ms on a single RTX 3080.

Hardware Requirements

VRAM for inference:

• The model checkpoint (≈ 1.2 GB in safetensors format) fits comfortably into 8 GB of GPU memory.
• For batch‑size = 1, 6 GB of VRAM is sufficient; larger batches (e.g., 4‑image batches) benefit from 12 GB‑plus GPUs.

Recommended GPU specifications:

• Any NVIDIA GPU with ≥ 6 GB VRAM (RTX 2060, RTX 3060, GTX 1080 Ti) for modest throughput.
• For production‑scale pipelines, a 12 GB‑24 GB device (RTX 3080, RTX 4090, A100) enables parallel processing of multiple pages.

CPU & storage:

• CPU is only needed for image loading and post‑processing; a modern 4‑core processor (e.g., Intel i5‑12400) is adequate.
• Disk space: the model files plus a small cache (~ 2 GB) – a 10 GB SSD partition is more than enough.

Performance characteristics:

• Inference latency: ~ 120 ms per 800×1200 px image on RTX 3080 (FP16).
• Throughput scales linearly with batch size until VRAM saturation.

Use Cases

Primary intended applications:

• Automated PDF to HTML conversion where preserving layout semantics is critical.
• Enterprise document‑search engines that need to index by region type (e.g., “find all tables in contracts”).
• Scientific literature mining – extracting figures, formulas, and captions for meta‑analysis.
• Form processing – isolating checkboxes, key‑value pairs, and signatures for RPA workflows.

Real‑world examples:

• A legal‑tech startup uses heron to separate clause headings from body text, enabling clause‑level search in thousands of contracts.
• Academic publishers run the model on incoming manuscripts to auto‑populate XML metadata for figures and tables.
• Finance departments feed scanned annual reports through the model to extract tables of financial statements for downstream analytics.

Integration possibilities:

• Directly via the transformers Python API (see the code snippet in the README).
• As a micro‑service behind a REST endpoint, using TorchServe or FastAPI.
• Bundled inside the Q4KM hard‑drive solution for offline, edge‑device deployment.

Training Details

Methodology:

• The model was trained using the RTDetrV2ForObjectDetection class from Hugging Face’s transformers library.
• Training employed mixed‑precision (FP16) on NVIDIA A100 GPUs, with a batch size of 16 images per GPU.
• Optimization used AdamW with a cosine‑annealed learning‑rate schedule, starting at 1e‑4 and decaying to 1e‑6 over 150 k steps.

Datasets:

• A curated mix of publicly available corpora (PubMed Central, arXiv, InvoiceNet) and proprietary Docling‑annotated documents, totaling ~ 2 M annotated pages.
• Annotations follow a 17‑class taxonomy, with each page containing an average of 12‑15 bounding‑box objects.

Compute requirements:

• Training was completed on a 4‑GPU A100 cluster (40 GB VRAM each) in roughly 48 hours.
• Fine‑tuning on a domain‑specific subset (e.g., medical forms) can be done on a single RTX 3090 in under 4 hours, using the same code base.

Related Papers

The model is grounded in two key publications:

• Advanced Layout Analysis Models for Docling (Livathinos et al., 2025) – introduces the heron architecture, training regimen, and benchmark suite.
• Docling Technical Report (Deep Search Team, 2024) – provides the broader context of the Docling pipeline, dataset construction, and evaluation methodology.

Both papers describe the use of the RT‑DETR‑v2 transformer backbone, the Hungarian matching loss, and the multi‑domain dataset that spans scientific, financial, and legal documents.

Licensing Information

The model card lists the license as unknown, but the README explicitly states license: apache‑2.0. In practice, this means the model weights and associated code are distributed under the permissive Apache 2.0 license, which: :

• Free commercial and non‑commercial use.
• Permission to modify, distribute, and create derivative works.
• Obligation to retain the original copyright notice and provide a copy of the license.
• No warranty; the user assumes all risk.

If a downstream project wishes to embed the model in a proprietary product, it may do so as long as the Apache 2.0 attribution notice is included in the documentation or “About” page. No additional royalties or source‑code disclosures are required.