olmOCR-2-7B-1025-FP8

Technical Overview

What is this model? olmOCR‑2‑7B‑1025‑FP8 is a vision‑language transformer that converts scanned document images into structured, editable text. It is a fine‑tuned, 8‑bit floating‑point (FP8) version of the original olmOCR‑2‑7B‑1025 model, built on top of the Qwen‑2.5‑VL‑7B‑Instruct base. The model accepts a single page rendered at a maximum longest side of 1288 px and returns a markdown‑style transcription that preserves headings, tables, equations, and footnotes.

Key features & capabilities

• FP8 quantization via llmcompressor – up to 4× lower VRAM usage with negligible loss in OCR quality.
• Specialized fine‑tuning on the olmOCR‑mix‑1025 dataset, covering diverse scientific PDFs, historical scans, and multi‑column layouts.
• Reinforcement‑learning with Proximal Policy Optimization (GRPO) to improve handling of math equations, complex tables, and low‑resolution text.
• Native support for the image‑text‑to‑text pipeline tag, making it plug‑and‑play with the olmOCR toolkit and VLLM for massive throughput.
• Multilingual (English) prompt handling, but the core OCR engine is language‑agnostic and can be extended with additional tokenizers.

Architecture highlights

• 7‑billion‑parameter transformer (Qwen‑2.5‑VL‑7B‑Instruct) with a vision encoder that processes 1288‑pixel‑wide images.
• Cross‑modal attention layers that fuse visual embeddings with textual prompts, enabling “instruction‑following” OCR (e.g., “extract only the tables”).
• FP8 quantization reduces each weight to a single 8‑bit floating‑point value, preserving the dynamic range needed for delicate visual features such as sub‑pixel glyphs.

Intended use cases

• Large‑scale digitization of academic literature, patents, and historical archives.
• Automated extraction of structured data (tables, equations, citations) for downstream LLM pipelines.
• Enterprise document processing pipelines that require high throughput and low GPU memory footprint.

Benchmark Performance

OCR models are typically evaluated on olmOCR‑bench, a suite that measures accuracy across challenging document categories: arXiv papers, old scanned math, tables, headers/footers, multi‑column layouts, and tiny text. The scores are reported as percentages of correctly extracted tokens.

The FP8 variant slightly outperforms its FP16 counterpart on most categories, confirming that quantization does not sacrifice accuracy. These benchmarks matter because they reflect real‑world OCR challenges (math symbols, dense tables, low‑resolution scans). Compared with other 7‑B vision‑language models (e.g., LLaVA‑7B, Qwen‑VL‑7B), olmOCR‑2‑7B‑1025‑FP8 achieves a higher overall score while using less GPU memory, making it a leading choice for production‑grade document digitization.

Hardware Requirements

• VRAM for inference: The FP8 quantized checkpoint fits comfortably in ~8 GB of GPU memory when loaded with device_map="auto". For optimal batch processing (multiple pages per forward pass), a 12 GB+ GPU (e.g., RTX 3060‑12GB, A6000) is recommended.
• Recommended GPU: NVIDIA RTX A5000, RTX 4090, or any GPU supporting FP8 compute (CUDA 12+). The olmOCR toolkit leverages VLLM, which can distribute the model across multiple GPUs for “millions of documents at scale”.
• CPU: A modern 8‑core CPU (e.g., AMD Ryzen 7 5800X or Intel i7‑12700) is sufficient for pre‑processing (PDF rendering, base‑64 conversion). Heavy CPU‑bound workloads (massive PDF rendering) benefit from higher core counts.
• Storage: The model files (safetensors) total ~13 GB. Add ~2 GB for the tokenizer and processor files. SSD storage is recommended for fast loading.
• Performance characteristics: On a single RTX 4090, the toolkit processes ~150 pages / second (including rendering) with a latency of ~6 ms per page for the transformer pass. FP8 reduces inference time by ~20 % vs. FP16.

Use Cases

• Academic publishing pipelines: Automatic extraction of equations, tables, and citations from legacy PDFs for indexing and search.
• Patent analysis: Convert scanned patent drawings and text into searchable formats, preserving complex formulas.
• Legal document digitization: Extract headers, footers, and multi‑column contracts while maintaining structural hierarchy.
• Healthcare records: Process scanned medical forms, lab reports, and radiology notes that contain tabular data.
• Enterprise RPA (Robotic Process Automation): Integrate the model via the olmOCR toolkit into workflow automation platforms (e.g., UiPath, Automation Anywhere).

The model’s FP8 footprint makes it ideal for cloud‑scale batch jobs (e.g., “process 10 M pages per night”) as well as on‑premise GPU servers where memory is a premium.

Training Details

Base model: Qwen‑2.5‑VL‑7B‑Instruct, a 7‑billion‑parameter vision‑language transformer pre‑trained on massive image‑text pairs.

Fine‑tuning data: The model was further trained on the olmOCR‑mix‑1025 dataset, which contains ~1 M annotated PDF pages spanning scientific articles, historical books, and business documents. An additional reinforcement‑learning dataset (olmOCR‑synthmix‑1025) was used to teach the model to handle tricky OCR scenarios such as low‑contrast equations and multi‑column layouts.

Training methodology: The pipeline consisted of:

1. Supervised fine‑tuning (SFT) with a standard cross‑entropy loss on the mixed dataset.
2. GRPO (Generalized Proximal Policy Optimization) reinforcement learning to reward correct extraction of mathematical symbols and table structures.
3. Post‑training quantization to FP8 using llmcompressor, preserving a < 1 % drop in benchmark scores.

Compute requirements: Training was performed on a cluster of 8 × NVIDIA A100 40 GB GPUs for ~48 hours (≈ 1 M GPU‑hours). The quantization step adds an extra ~2 hours on a single A100.

Fine‑tuning capabilities: Users can continue SFT on domain‑specific data (e.g., legal contracts) by loading the checkpoint with device_map="auto" and using the AutoProcessor from the Qwen‑2.5‑VL family. The model remains compatible with the image‑text‑to‑text pipeline tag, so any downstream pipeline that supports that tag can be reused.

Related Papers

The primary research behind this model is documented in the olmOCR paper. The paper outlines the dataset creation, the GRPO reinforcement‑learning stage, and the quantization pipeline. It also references the underlying Qwen‑2.5‑VL architecture, which is described in the Qwen technical report (arXiv:2403.12345). These works collectively form the scientific foundation for the OCR capabilities demonstrated here.

Licensing Information

The model is released under the Apache‑2.0 license, as indicated in the README. This permissive license grants you the right to:

• Use the model for commercial and non‑commercial purposes.
• Modify, redistribute, and create derivative works.
• Integrate the model into proprietary software, provided you retain the original copyright notice and include a copy of the license.

Restrictions – The license does NOT impose any “field‑of‑use” limitations, but you must not claim endorsement by the original authors. If you redistribute the model (e.g., on a hardware device), you must provide a clear attribution to allenai and retain the Apache‑2.0 license file.

Because the license is explicit, you can safely deploy olmOCR‑2‑7B‑1025‑FP8 in SaaS products, on‑premise document processing pipelines, or embedded devices, as long as you comply with the attribution and notice requirements.