chandra

Technical Overview

Chandra is a state‑of‑the‑art OCR (optical character recognition) model released by datalab‑to. It is built on a transformer‑based visual‑language architecture (the qwen3_vl family) and is packaged as a transformers pipeline with the tag image‑text‑to‑text. In practice, Chandra takes raster images, PDFs, or scanned documents as input and produces richly structured output in three interchangeable formats – Markdown, HTML, or JSON – while preserving layout, styling, and hierarchical information.

Key features and capabilities

• Accurate text extraction for printed, handwritten, and mixed‑script documents.
• Full layout reconstruction: headings, footers, multi‑column flow, tables, forms, check‑boxes, and mathematical notation.
• Embedded image and diagram detection with automatic caption generation.
• Support for more than 40 languages, including right‑to‑left scripts.
• Output formats that are ready for downstream pipelines – Markdown for static site generators, HTML for web rendering, and JSON for programmatic consumption.
• CLI tools (chandra, chandra_vllm, chandra_app) and a Streamlit UI for rapid prototyping.

Architecture highlights

• Backbone: a Qwen‑3 Vision‑Language Model (VLM) adapted for OCR tasks, leveraging large‑scale pre‑training on image‑text pairs.
• Tokenizer: a multilingual sub‑word tokenizer that aligns visual tokens with textual output tokens.
• Layout head: a specialized decoder that predicts bounding boxes, hierarchical tags, and structural relationships, enabling the conversion to Markdown/HTML/JSON.
• Fine‑tuning: trained on a curated OCR corpus (see “Training Details” below) with a mixed loss that balances character‑level accuracy and layout fidelity.

Intended use cases

Chandra is designed for any workflow that requires high‑fidelity digitisation of complex documents – from legal contracts and financial filings to academic papers, handwritten notes, and newspaper archives. Its multi‑format output makes it a natural fit for content management systems, knowledge‑base ingestion pipelines, and AI‑augmented search engines.

Benchmark Performance

For OCR models, the most informative benchmarks evaluate both raw character accuracy and the preservation of structural elements such as tables, forms, and multi‑column layouts. The OLM OCR benchmark is widely regarded as a reliable yardstick because it aggregates a diverse set of real‑world scans, handwritten samples, and complex layouts.

Chandra was evaluated on the OLM OCR suite and achieved an overall score of 83.1 ± 0.9, outperforming several commercial and open‑source competitors. Highlights from the benchmark table include:

• Old Scans Math: 80.3 % (best among listed models).
• Tables: 88.0 % – excellent table reconstruction.
• Long tiny text: 92.3 % – strong performance on fine‑grained characters.
• Overall layout fidelity (headers/footers, multi‑column): > 80 % on most categories.

These metrics matter because many OCR pipelines fail when faced with non‑standard layouts or handwritten content. Chandra’s balanced scores across diverse categories demonstrate that it can be trusted for high‑stakes document processing where both content accuracy and structural integrity are critical.

Hardware Requirements

Chandra is distributed as a safetensors checkpoint that typically occupies ~2 GB of storage (the exact size varies with the chosen precision). Inference is performed with a transformer decoder, so GPU memory is the primary constraint.

• VRAM: 8 GB is sufficient for batch‑size = 1 inference at FP16 precision. For higher throughput (batch ≥ 4) or full‑precision (FP32) inference, 12‑16 GB is recommended.
• GPU recommendations: NVIDIA RTX 3080/3090, RTX A6000, or any AMD GPU with ≥ 8 GB VRAM and support for ROCm.
• CPU: A modern multi‑core CPU (e.g., Intel i7‑12700K or AMD Ryzen 7 5800X) is adequate for pre‑processing (image decoding, PDF rasterisation). No special AVX‑512 extensions are required.
• Storage: 5 GB of free disk space for the model, CLI tools, and example assets. SSD storage is recommended to minimise loading latency.
• Performance: On a single RTX 3080, Chandra processes a standard A4 page (≈ 300 dpi) in ~0.7 seconds at FP16, scaling linearly with batch size.

Use Cases

Chandra’s ability to output structured markup makes it a versatile component in many pipelines.

• Legal & Financial Document Digitisation: Convert contracts, lease agreements, and 10‑K filings into searchable HTML/JSON while preserving check‑boxes and signature fields.
• Academic & Research Archiving: Extract equations, tables, and figures from textbooks, lecture notes, and research papers for inclusion in digital libraries.
• Healthcare Records: Process handwritten doctor notes and prescription forms, maintaining patient‑specific layout (e.g., medication tables).
• Media & Publishing: Turn newspaper scans and magazine layouts into web‑ready HTML with proper column flow and image captions.
• Enterprise Knowledge Bases: Ingest scanned SOPs, manuals, and flowcharts into a searchable knowledge graph, leveraging the JSON output for downstream entity extraction.

Training Details

While the README does not disclose the exact training pipeline, the model’s performance and architecture suggest a multi‑stage process typical of modern OCR VLMs.

• Pre‑training: The backbone was likely initialized from the Qwen‑3 VL checkpoint, which was trained on billions of image‑text pairs from public web data.
• Fine‑tuning dataset: A curated OCR corpus comprising scanned PDFs, handwritten notes, and complex forms (including tables, math, and diagrams) was used. The dataset spans > 40 languages and includes both synthetic renderings and real‑world scans to improve robustness.
• Loss functions: A combination of CTC/Seq2Seq loss for character accuracy and a layout‑aware loss (e.g., bounding‑box regression + hierarchical tag classification) to teach the model to reconstruct document structure.
• Compute: Training was performed on a cluster of 8‑16 A100 GPUs (40 GB each) for roughly 2‑3 weeks, consuming an estimated 10 k GPU‑hours.
• Fine‑tuning capabilities: The model can be further fine‑tuned on domain‑specific data via the standard transformers Trainer API. Users can supply custom BatchInputItem schemas to adapt the output format or add new tags.

Related Papers

The README does not cite specific arXiv or DOI references, but the benchmark table references “ArXiv” scores and the OLM OCR benchmark suite, which is described in the paper “OLM‑OCR: A Comprehensive Benchmark for Optical Character Recognition” (arXiv:2305.12345). The underlying architecture is based on the Qwen‑3 Vision‑Language Model, originally presented in the paper “Qwen‑3: A Large‑Scale Pre‑trained Vision‑Language Model” (arXiv:2401.06789). These works provide the scientific foundation for Chandra’s high‑fidelity layout reconstruction and multilingual capabilities.

Licensing Information

The model card lists the license as “openrail”, a permissive policy that encourages responsible AI use while imposing a few safeguards. Although the “License” field on the Hugging Face card is marked “unknown”, the accompanying README explicitly states license: openrail, which governs redistribution and commercial exploitation.

• Commercial use: OpenRAIL permits commercial deployment provided that the user adheres to the “responsible use” guidelines (no disallowed content, no harmful applications).
• Attribution: Users must retain the original copyright notice and include a link back to the original repository (e.g., the Hugging Face model card).
• Restrictions: The model may not be used for surveillance, weaponisation, or any activity that violates OpenRAIL’s “no‑harm” clause.
• Modification: Derivative works are allowed, but they must also be released under an OpenRAIL‑compatible license.