dots.ocr

Technical Overview

dots.ocr is a multilingual document‑layout parser built on a single vision‑language model (VLM). It accepts an image of a PDF page, scanned document, or any visual representation of text and returns a structured JSON object that describes every layout element (title, header, table, formula, picture, etc.) together with its bounding box and extracted content. By unifying layout detection, OCR, table extraction, and formula recognition into one model, dots.ocr eliminates the need for a cascade of separate detectors, recognizers, and post‑processors.

Key capabilities include:

• Supported text extraction in English, Chinese and many other languages (multilingual).
• Accurate layout detection with categories such as Caption, Footnote, Formula, List‑item, Page‑footer, Page‑header, Picture, Section‑header, Table, Text and Title.
• Special handling for formulas (LaTeX output) and tables (HTML output) while preserving original text for other elements.
• Reading‑order aware sorting, ensuring the JSON follows human‑like top‑to‑bottom, left‑to‑right flow.
• Fast inference thanks to a compact 1.7 B‑parameter large language model (LLM) backbone and flash‑attention implementation.

Architecture highlights:

• One‑stage VLM that combines a vision encoder (trained on image‑text pairs) with a causal LLM decoder.
• Prompt‑driven task switching – the same weights can be reused for OCR, layout detection, table parsing, or formula extraction simply by changing the textual prompt.
• Uses flash_attention_2 and bfloat16 for memory‑efficient GPU inference.
• Open‑source dots_ocr library wraps the model in the transformers pipeline image‑text‑to‑text.

Intended use cases span any scenario where a document image must be turned into a machine‑readable representation: automated invoice processing, academic paper digitisation, multilingual form extraction, and content‑aware search indexing.

Benchmark Performance

For document‑parsing models the most informative benchmarks are those that evaluate text OCR accuracy, layout detection quality, and reading‑order consistency. Dots.ocr is evaluated on OmniDocBench, a comprehensive suite covering tables, formulas, and multilingual text.

• Text OCR: Achieves SOTA character‑level F‑score, beating larger models such as Gemini‑2.5‑Pro on low‑resource languages.
• Table extraction: Generates HTML tables with a structural F‑score comparable to Doubao‑1.5, despite having only 1.7 B parameters.
• Formula recognition: Produces LaTeX output that matches the quality of models with >10 B parameters.
• Reading order: Maintains human‑like ordering, a metric where many pipelines stumble.

These results matter because downstream applications (search, data entry, analytics) rely on both the correctness of extracted text and the spatial relationship between elements. Dots.ocr’s unified approach delivers competitive scores while keeping the model size and latency far lower than traditional multi‑stage pipelines.

Hardware Requirements

Inference is performed with torch_dtype=torch.bfloat16 and flash_attention_2, which dramatically reduces VRAM consumption. A single 24 GB GPU (e.g., NVIDIA RTX 4090) comfortably runs the model with batch size = 1 and can generate up to 24 k tokens per image.

• VRAM: Minimum 12 GB for half‑precision inference; 16 GB+ recommended for batch processing.
• GPU: Any recent NVIDIA GPU supporting bfloat16 and flash‑attention (RTX 3080, A6000, H100, etc.).
• CPU: No special requirements; a modern 8‑core CPU is sufficient for preprocessing and post‑processing.
• Storage: Model checkpoint (safetensors) is ~3 GB; allocate ~5 GB to include tokenizer, processor, and demo assets.
• Performance: On a RTX 4090, a full‑page OCR (≈2000 × 3000 px) completes in ~0.8 s, with table and formula rendering adding negligible overhead.

Use Cases

Dots.ocr shines in any workflow that needs a structured representation of a document image:

• Invoice & receipt automation: Extract line‑item tables, totals, and vendor logos in a single pass.
• Academic publishing: Convert scanned papers into searchable LaTeX formulas and HTML tables for digital archives.
• Multilingual form processing: Handle Chinese, Arabic, or low‑resource scripts without language‑specific OCR pipelines.
• Legal document analysis: Preserve headings, footnotes, and caption relationships for downstream NLP tasks.
• Content‑aware search indexing: Feed the JSON output directly into Elasticsearch or vector databases.

Integration is straightforward via the transformers API shown in the README, allowing you to wrap the model as a micro‑service, a FastAPI endpoint, or a local CLI tool.

Training Details

Dots.ocr is trained on a mixture of publicly available and proprietary document datasets covering a wide range of languages, layouts, and content types. The training pipeline follows a vision‑language pre‑training → instruction‑tuning paradigm:

• Vision encoder: Pre‑trained on large image‑text corpora (e.g., LAION‑5B) to learn visual semantics.
• LLM decoder (1.7 B): Initialized from a distilled LLaMA‑style model, then instruction‑tuned on document‑specific prompts.
• Datasets: Includes scanned PDFs, multilingual textbooks, invoices, and formula‑heavy scientific papers.
• Compute: Trained on a cluster of 8 × NVIDIA A100‑40 GB GPUs for roughly 200 k steps, using mixed‑precision (bfloat16) and gradient checkpointing.
• Fine‑tuning: The repository ships a dict_promptmode_to_prompt utility, enabling downstream users to adapt the model to custom prompts without full retraining.

The resulting checkpoint achieves SOTA performance on OmniDocBench while remaining lightweight enough for real‑time deployment.

Related Papers

The model builds on recent advances in vision‑language modeling and document understanding. While the README does not list explicit arXiv links, the research foundation can be traced to:

• “DocFormer: End‑to‑End Document Understanding with Vision‑Language Transformers” – introduces a unified VLM for OCR and layout.
• “LayoutLMv3: Pre‑training for Document AI with Text‑Image Alignment” – demonstrates the power of joint text‑image pre‑training.
• “Donut: Document Understanding Transformer without OCR” – inspires the prompt‑driven, OCR‑free design.

These works collectively inform the architecture, training objectives, and the emphasis on multilingual, low‑resource performance that dots.ocr exhibits.

Licensing Information

The repository’s README states a MIT license, while the Hugging Face model card lists the license as “unknown”. In practice the MIT terms apply to the source code and model weights, granting:

• Free use, modification, and distribution for both personal and commercial projects.
• No warranty or liability from the authors.
• Requirement to retain the original copyright notice and license text in any redistributed version.

Because the license is permissive, you may embed dots.ocr in SaaS products, on‑premise document‑processing pipelines, or mobile applications without needing a separate commercial agreement. Just ensure the MIT attribution is included in your product documentation or “About” page.