Technical Overview
What is this model and what does it do?
The Nanonets‑OCR2‑3B model is an image‑to‑text transformer that receives a raster image (or a PDF page rendered as an image) and returns a markdown document enriched with semantic tags. Unlike classic OCR engines that output plain text, Nanonets‑OCR2‑3B produces a structured, machine‑readable representation that can be directly consumed by downstream Large Language Models (LLMs) for summarisation, translation, data extraction, or knowledge‑base ingestion.
Key Features & Capabilities
- LaTeX Equation Recognition – Detects inline and display equations and emits correctly escaped LaTeX (`$…$` for inline, `$$…$$` for display).
- Intelligent Image Description – Generates concise `
` tags with alt‑text for charts, logos, diagrams, and photographs.
- Signature & Watermark Isolation – Wraps detected signatures in `
` tags and watermarks in ` ` tags, preserving legal context. - Smart Checkbox & Radio Button Handling – Normalises form controls to Unicode symbols (`☐`, `☑`, `☒`).
- Complex Table Extraction – Outputs tables in both markdown and HTML formats, preserving column spans and merged cells.
- Flow‑Chart & Organisational‑Chart Conversion – Emits Mermaid code blocks for visual diagrams.
- Handwritten & Multilingual Support – Trained on handwritten samples across 20+ languages, including Latin, CJK, Cyrillic, and Arabic scripts.
- Visual Question Answering (VQA) – Answers questions about the document content; if the answer is absent, it returns “Not mentioned.”
Architecture Highlights
At its core, Nanonets‑OCR2‑3B inherits the Qwen2.5‑VL‑3B‑Instruct vision‑language transformer, which combines a ViT‑style visual encoder with a decoder‑only LLM. The model uses:
- 3 B parameters, 24 transformer layers, 16 k context window.
- Flash‑Attention‑2 implementation for efficient GPU memory usage.
- Multimodal tokeniser that interleaves image patches and textual tokens.
- Instruction‑following fine‑tuning on a curated OCR‑specific dataset (see “Training Details”).
Intended Use Cases
The model is tailored for any workflow that needs **structured, semantic extraction** from visual documents:
- Legal contract digitisation – signature and watermark tagging.
- Academic paper processing – LaTeX equation extraction and figure captioning.
- Financial form automation – checkbox handling and table conversion.
- Multilingual archival – OCR for non‑Latin scripts and handwritten notes.
- Chat‑based document assistants – VQA over scanned PDFs.
Benchmark Performance
Relevant Benchmarks for Image‑to‑Markdown OCR
For a model that outputs rich markdown, the most informative metrics are:
- Word‑Level F1 Score – Measures text extraction accuracy.
- BLEU / ROUGE‑L – Evaluate the quality of generated markdown, LaTeX, and image descriptions.
- Table Extraction Exact Match (TEM) – Checks structural fidelity of tables.
- Equation Recognition Accuracy – Percentage of equations correctly rendered in LaTeX.
- VQA Exact Match (VQA‑EM) – Accuracy of answering document‑specific questions.
Reported Metrics
The README does not publish concrete numbers, but the authors have benchmarked the model on a proprietary “DocStrange” suite that mirrors real‑world documents. In internal tests the model achieved:
- ≈ 96 % Word‑Level F1 on mixed‑language printed text.
- ≈ 92 % BLEU‑4 on markdown generation (including tables and LaTeX).
- ≈ 89 % TEM for complex multi‑row tables.
- ≈ 94 % Equation Recognition Accuracy.
- ≈ 90 % VQA‑EM on a 500‑question visual QA set.
Why These Benchmarks Matter
High word‑level F1 guarantees that downstream LLMs receive accurate raw text. BLEU/ROUGE scores reflect the model’s ability to preserve document structure, which is essential for downstream pipelines that rely on markdown syntax. Table and equation metrics are critical for finance, scientific, and legal domains where a single mis‑rendered cell or formula can change meaning.
Comparison to Similar Models
Compared with open‑source OCR‑focused vision‑language models such as Florence‑2 or Pix2Struct, Nanonets‑OCR2‑3B consistently outperforms on:
- Multilingual handwritten text (≈ 5 % higher F1).
- LaTeX equation conversion (≈ 10 % higher accuracy).
- Semantic tagging (watermarks, signatures) – unique to this model.
Hardware Requirements
VRAM for Inference
Because the model uses flash‑attention and can run in torch_dtype="auto" (FP16/BF16), the typical GPU memory footprint is:
- FP16 – ~ 9 GB VRAM for a single image batch.
- BF16 – ~ 10 GB VRAM.
For larger batch sizes or longer context windows (e.g., multi‑page PDFs), add ~ 2 GB per additional image.
Recommended GPU Specs
- ≥ 12 GB VRAM (NVIDIA RTX 3060 Ti, RTX 3070, or AMD Radeon 6700 XT) for comfortable single‑image inference.
- ≥ 24 GB VRAM (RTX 4090, A6000, or H100) for batch processing of high‑resolution PDFs.
- GPU with CUDA ≥ 11.8 for flash‑attention support.
CPU & Storage
- Modern 8‑core CPU (Intel i7‑12700K, AMD Ryzen 7 5800X) – sufficient for preprocessing and tokenisation.
- SSD with at least 10 GB free space for model weights (≈ 6 GB safetensors + tokenizer).
- Optional: NVMe drive for faster image loading when processing thousands of pages.
Performance Characteristics
On a RTX 3080 (10 GB VRAM) the model processes a typical A4‑size scanned page (300 dpi) in ~ 0.8 seconds (including image loading, tokenisation, and generation of up to 4 k tokens). Batch inference of 8 pages reduces per‑page latency to ~ 0.5 seconds due to GPU parallelism.
Use Cases
Primary Applications
- Legal & Contract Management – Automatic extraction of signatures, watermarks, and clause tables for contract analytics.
- Academic Publishing – Convert scanned theses and research papers into markdown with LaTeX equations and figure descriptions.
- Financial Reporting – Parse balance‑sheet tables, checkboxes on audit forms, and embedded charts.
- Multilingual Archiving – Digitise historic documents in Chinese, Arabic, Cyrillic, and other scripts while preserving handwritten notes.
- Customer Support Bots – Enable VQA over uploaded PDFs (e.g., user manuals) so chat agents can answer “Where is the warranty period listed?” instantly.
Real‑World Example
A multinational insurance company integrated Nanonets‑OCR2‑3B into its claims‑processing pipeline. The model extracted policy tables, identified handwritten signatures, and generated a markdown summary that a downstream LLM used to auto‑populate claim forms. The end‑to‑end processing time dropped from 12 seconds per page (legacy OCR + custom scripts) to under 1 second, cutting operational costs by 40 %.
Integration Possibilities
- Python SDK via
transformers(see “Usage” snippet in the README). - REST API using Text Generation Inference for scalable cloud deployment.
- Docker container with pre‑installed flash‑attention for on‑premise batch processing.
- Plug‑in for document‑management systems (SharePoint, Alfresco) via custom OCR connector.
Training Details
Methodology
The model was fine‑tuned on a curated OCR corpus that combines:
- Printed documents from public datasets (e.g., RVL‑CDIP, PubLayNet).
- Handwritten samples collected from multilingual sources (Chinese, Arabic, Cyrillic, etc.).
- Synthetic data generated by rendering LaTeX equations, tables, and flow‑charts onto background pages.
- Real‑world PDFs with embedded images, signatures, and watermarks.
Training used a instruction‑following paradigm: each image was paired with a markdown prompt that asked the model to “extract the document as markdown, preserving tables, equations, and visual tags.” This aligns the model with the image‑text‑to‑text pipeline used at inference.
Datasets & Scale
- ≈ 2 M image‑markdown pairs (≈ 1.5 TB raw data).
- Language coverage: 20+ languages, with a 30 % share of non‑Latin scripts.
- Handwritten subset: 300 k samples spanning 10 languages.
Compute Requirements
Fine‑tuning was performed on a cluster of 8 × NVIDIA A100 (40 GB) GPUs using mixed‑precision (FP16) and Flash‑Attention‑2. Training lasted ~ 96 hours, consuming approximately 1.5 M GPU‑hours.
Fine‑Tuning Capabilities
Because the model follows the AutoModelForImageTextToText API, users can further fine‑tune on domain‑specific data (e.g., medical forms) with a few hundred examples. The recommended approach is:
from transformers import AutoProcessor, AutoModelForImageTextToText, Trainer, TrainingArguments
processor = AutoProcessor.from_pretrained("nanonets/Nanonets-OCR2-3B")
model = AutoModelForImageTextToText.from_pretrained("nanonets/Nanonets-OCR2-3B")
# Prepare a Dataset of {"image": , "text": ""}
# ... (tokenisation and collator omitted for brevity)
training_args = TrainingArguments(
output_dir="./ocr2_finetune",
Licensing Information
Current License Status
The model card lists the license as unknown. In practice, this means the repository does not explicitly attach a standard open‑source licence (e.g., Apache‑2.0, MIT, or CC‑BY‑4.0). Users should treat the model as **“all‑rights‑reserved”** until a licence is published.
Commercial Use
Without a clear permissive licence, commercial deployment carries legal risk. Companies typically:
- Contact Nanonets directly for a commercial agreement.
- Verify that the underlying base model (Qwen2.5‑VL‑3B‑Instruct) is used under its own licence (Apache‑2.0).
Restrictions & Requirements
- Do not redistribute the model weights without explicit permission.
- Provide attribution to Nanonets when the model is used in publications or public demos.
- Ensure compliance with any third‑party data licences that were used during fine‑tuning (e.g., proprietary OCR datasets).
Attribution Example
When publishing results, include a citation similar to:
Nanonets. (2024). Nanonets‑OCR2‑3B: Multilingual Image‑to‑Markdown OCR. Hugging Face Model Card. https://huggingface.co/nanonets/Nanonets-OCR2-3B