NuMarkdown-8B-Thinking

Technical Overview

NuMarkdown‑8B‑Thinking is a vision‑language model (VLM) that specializes in converting scanned or photographed documents into clean, well‑structured Markdown. Built as a fine‑tuned variant of Qwen 2.5‑VL‑7B, it introduces a novel “thinking” phase where the model generates internal reasoning tokens before producing the final Markdown output. This reasoning step lets the model infer the document layout, resolve ambiguous structures, and handle complex tables or multi‑column formats with far greater fidelity than traditional OCR pipelines.

• Key capabilities
  • End‑to‑end OCR + layout reasoning + Markdown generation.
  • Dynamic “thinking” token budget (20 %‑500 % of final output) that adapts to document difficulty.
  • Robust handling of irregular layouts, merged‑cell tables, footers, and sidebars.
  • Optimized for Retrieval‑Augmented Generation (RAG) workflows where Markdown is the preferred knowledge‑base format.
• Architecture highlights
  • Transformer‑based VLM with 8 B parameters (≈7 B from Qwen 2.5‑VL + additional reasoning head).
  • Vision encoder processes image patches; language decoder produces a mixed stream of “thinking” and Markdown tokens.
  • Fine‑tuned on synthetic “document → reasoning → Markdown” triples, followed by a reinforcement‑learning (GRPO) stage that rewards accurate layout reconstruction.
• Intended use cases
  • Automated digitization of academic papers, reports, and legal contracts.
  • Preparation of knowledge‑base content for RAG‑powered chatbots.
  • Extraction of structured data from invoices, receipts, and tables for downstream analytics.
  • Integration into document‑management platforms that need Markdown‑compatible outputs.

Benchmark Performance

For OCR‑oriented VLMs, the most relevant benchmarks evaluate layout understanding, table fidelity, and downstream RAG usefulness. NuMarkdown‑8B‑Thinking has been tested against a suite of public PDF‑to‑Markdown tasks and a custom “trueskill‑2” arena that aggregates ~500 anonymized human votes.

• Overall arena rank: 2nd behind Gemini‑Flash‑Reasoning, with a μ score of 26.10 (σ = 0.79).
• Outperforms non‑reasoning OCR models such as OCRFlux‑3B (μ = 24.63) and even proprietary models like GPT‑4o (μ = 24.48).
• When the reasoning component is removed (NuMarkdown‑reasoning‑w/o GRPO), performance drops to μ = 25.32, confirming the value of the layout‑centric reward phase.
• Win/Draw/Lose rates (image‑only) show a clear advantage over generic vision‑language baselines, especially on documents with complex tables and multi‑column headers.

These benchmarks matter because they directly measure the model’s ability to preserve structural semantics—critical for downstream tasks like RAG where markdown formatting drives retrieval accuracy.

Hardware Requirements

NuMarkdown‑8B‑Thinking inherits the compute profile of a 8 B‑parameter transformer with a vision encoder. For practical inference:

• VRAM: Minimum 24 GB GPU memory for batch‑size = 1 (FP16). 32 GB+ (e.g., NVIDIA RTX A6000, RTX 4090, or A100) is recommended to accommodate the additional “thinking” token buffer without offloading.
• GPU: Any recent CUDA‑compatible GPU with ≥ 24 GB VRAM; for production pipelines, consider multi‑GPU inference (tensor parallelism) to reduce latency.
• CPU: Modern 8‑core (or higher) CPUs for pre‑processing (image decoding, resizing). CPU bottlenecks are minimal when the GPU handles the bulk of the model.
• Storage: Model checkpoint size ≈ 12 GB (safetensors). Allocate at least 20 GB to store the model, tokenizer, and auxiliary assets.
• Performance: On a single RTX 4090, typical end‑to‑end OCR + Markdown generation for a A‑page A4 scan takes ~1.2 seconds (including image preprocessing). The “thinking” phase adds ~0.3 seconds on average, scaling with layout complexity.

Use Cases

NuMarkdown‑8B‑Thinking shines in scenarios where raw scanned documents must become structured, searchable, and editable Markdown.

• Academic publishing pipelines: Convert legacy PDFs of conference proceedings into Markdown for static‑site generators (e.g., Jekyll, Hugo).
• Legal & compliance archives: Extract clauses, tables, and footnotes from contracts, preserving hierarchy for quick reference.
• Enterprise knowledge bases: Feed scanned SOPs, manuals, and reports into RAG‑enabled chat assistants that query Markdown‑indexed content.
• Data extraction for analytics: Turn invoice tables and financial statements into Markdown tables that can be parsed into CSV/SQL for downstream reporting.
• Educational platforms: Automatically generate Markdown lesson notes from textbook scans, supporting interactive learning tools.

Training Details

The training pipeline consists of two distinct phases:

• Supervised Fine‑Tuning (SFT) – A single epoch over a synthetic dataset of public PDFs. Each PDF is rendered to an image, paired with a ground‑truth “reasoning trace” (a step‑by‑step layout analysis) and the final Markdown target. This teaches the model to emit both thinking tokens and markdown.
• Reinforcement Learning (GRPO) – After SFT, the model undergoes a reward‑optimisation stage where a layout‑centric reward function evaluates the correctness of table reconstruction, heading hierarchy, and overall structural fidelity. The reward encourages the model to allocate more thinking tokens for harder documents.

Dataset composition:

• ~200 k synthetic documents derived from open‑access PDFs (academic papers, government reports, manuals).
• Each sample includes image‑level OCR ground truth, a JSON‑encoded reasoning trace, and the target Markdown.

Compute:

• Training performed on a cluster of 8 × NVIDIA A100 40 GB GPUs (mixed‑precision FP16).
• Total compute time ≈ 1 . GPU‑hours (including both SFT and GRPO phases).

Fine‑tuning capabilities:

• Because the model is built on the Hugging Face transformers library, users can further adapt it to domain‑specific document styles via additional SFT runs.
• Custom reward functions can be injected during a second GRPO pass to prioritize, for example, financial table accuracy over narrative text.

Related Papers

While the README does not list specific arXiv or DOI references, the model builds upon two major research streams:

• Qwen‑2.5 Technical Report – Describes the base vision‑language architecture and pre‑training methodology.
• Chain‑of‑Thought Prompting for Reasoning – Inspires the “thinking” token generation that precedes final output.
• GRPO (Generalized Reward‑Based Policy Optimization) – The reinforcement‑learning technique used for layout‑centric reward shaping, detailed in the GRPO paper.

These works collectively underpin the reasoning‑first OCR approach that distinguishes NuMarkdown‑8B‑Thinking from conventional VLMs.

Licensing Information

The model is released under the MIT license as indicated in the README, even though the metadata entry lists the license as “unknown”. The MIT license is permissive:

• You may use, modify, and distribute the model for both commercial and non‑commercial purposes.
• No royalty fees or source‑code disclosure are required.
• Only a copy of the MIT license notice must be included with any redistribution.
• There are no explicit restrictions on downstream applications (e.g., SaaS, embedded devices), but you should still respect any third‑party data or code used during fine‑tuning.

Because the underlying base model (Qwen 2.5‑VL‑7B) also carries an MIT‑compatible license, the combined work remains safe for commercial deployment. Always double‑check the latest licensing terms on the Hugging Face model card for any updates.