GLM-OCR

Technical Overview

GLM‑OCR (model ID zai-org/GLM-OCR) is a multimodal optical‑character‑recognition (OCR) system built on the GLM‑V encoder‑decoder framework. It accepts raster images (or PDF pages rendered as images) and produces natural‑language text in up to eight languages (Chinese, English, French, Spanish, Russian, German, Japanese, Korean). The model is designed for “complex document understanding” – not only plain text lines but also tables, formulas, seals, code snippets and other layout‑rich elements.

Key capabilities include:

• Multi‑Token Prediction (MTP) loss – predicts several tokens in parallel, dramatically speeding up decoding without sacrificing accuracy.
• Stable full‑task reinforcement learning – jointly optimises layout analysis and text generation, yielding better generalisation on out‑of‑distribution documents.
• Lightweight cross‑modal connector – efficiently downsamples visual token streams before feeding them to the language decoder, keeping the total parameter count under 0.9 B.
• End‑to‑end pipeline – combines the CogViT visual encoder, a PP‑DocLayout‑V3 layout analyser, and a GLM‑0.5 B language decoder for parallel recognition.

Architecture highlights:

• Visual front‑end: CogViT pre‑trained on massive image‑text corpora.
• Cross‑modal bridge: token‑downsampling layer that reduces the visual sequence length while preserving spatial cues.
• Language back‑end: GLM‑0.5 B decoder (a stripped‑down version of the GLM‑2 family) that handles multilingual text generation.
• Two‑stage inference: first a layout detector (PP‑DocLayout‑V3) isolates text blocks, then the OCR engine transcribes each block in parallel.

Intended use cases span any scenario where high‑fidelity text extraction from scanned or photographed documents is required – from invoice processing to academic paper digitisation, and from legal contract analysis to multilingual archival retrieval.

Benchmark Performance

GLM‑OCR’s results are reported on several industry‑standard document‑understanding suites. The most notable figure is a 94.62 % overall score on OmniDocBench V1.5, placing it at the top of the leaderboard at the time of release. Additional benchmarks include:

• Formula recognition, table extraction, and information‑extraction tasks – all reported as “state‑of‑the‑art” in the README.
• Speed tests on a single‑GPU, single‑concurrency setup: 1.86 pages / second for PDFs and 0.67 images / second, outpacing comparable open‑source OCR models.

These benchmarks matter because they combine accuracy (how many characters, cells or fields are correctly recovered) with throughput (how fast a production‑scale pipeline can process documents). GLM‑OCR’s balance of high precision and low latency makes it attractive for real‑time or high‑volume batch processing, where many alternatives either lag in quality (e.g., generic scene‑text models) or in speed (large‑scale transformer OCRs).

Hardware Requirements

GLM‑OCR runs efficiently thanks to its sub‑billion‑parameter footprint, but inference still benefits from modern GPU hardware.

• VRAM: Minimum 12 GB for a single‑image batch; 16 GB+ recommended for batch processing of PDFs (multiple pages per request).
• GPU recommendation: NVIDIA RTX 3080/3090, RTX 4090, or comparable AMD Instinct GPUs. The model is compatible with vLLM, SGLang and Ollama, which can automatically shard the model across multiple GPUs if larger batch sizes are needed.
• CPU: A recent multi‑core CPU (e.g., Intel Xeon E5‑2690 v4 or AMD Ryzen 7 5800X) for pre‑processing (image loading, PDF rasterisation) and for feeding the GPU. CPU‑only inference is possible but will be dramatically slower (≈0.1 pages / second).
• Storage: The model checkpoint (including safetensors) occupies roughly 3 GB. Adding the SDK, layout analyser and example data brings total disk usage to ~5 GB.
• Performance characteristics: With the recommended GPU, latency per image is ~0.5 s, and per PDF page ~0.3 s when using the vLLM server with 8‑bit quantisation.

Use Cases

GLM‑OCR shines in any workflow that requires accurate extraction of structured text from scanned or photographed documents.

• Financial services: Automatic invoice, receipt and statement parsing, including table‑based line items and handwritten signatures.
• Legal & compliance: Extraction of clauses, seals, and table‑formatted data from contracts, court filings and regulatory PDFs.
• Academic & scientific publishing: Recognition of formulas, code snippets and multilingual abstracts from research papers.
• Government & public sector: Digitisation of multilingual forms, passports, and archival records with complex layouts.
• Enterprise document management: Integration into RPA pipelines via the official SDK, enabling end‑to‑end document intelligence (layout → structured JSON → downstream analytics).

The model can be called directly via the Transformers API, through vLLM/SGLang servers, or via the Ollama CLI, making it easy to embed in Python services, micro‑services, or even edge devices with sufficient GPU memory.

Training Details

GLM‑OCR was trained using a combination of supervised OCR data and multimodal image‑text pre‑training:

• Datasets: Large‑scale synthetic document corpora, real‑world scanned PDFs (e.g., PubLayNet, DocBank), and multilingual OCR benchmarks (ICDAR‑MHT, RRC‑MLT). The visual encoder was pre‑trained on CLIP‑style image‑text pairs before being fine‑tuned on document‑specific data.
• Losses: The novel Multi‑Token Prediction (MTP) loss enables parallel token generation, while a stable full‑task reinforcement learning (RL) objective aligns layout detection and text generation.
• Compute: Training was performed on a cluster of 8 × NVIDIA A100‑40 GB GPUs for roughly 2 weeks (≈10 k GPU‑hours). Mixed‑precision (FP16) and gradient checkpointing were used to keep memory usage under 30 GB per GPU.
• Fine‑tuning: The model can be further fine‑tuned on domain‑specific corpora (e.g., medical forms, legal contracts) using the same MTP loss. The official SDK provides scripts for continued training with the transformers library.

Related Papers

While the README does not list explicit citations, GLM‑OCR builds upon several well‑known research works:

• CogViT: Coarse‑to‑Fine Vision‑Language Pre‑training – the visual encoder backbone.
• GLM‑V: General Language Model for Vision‑Language Tasks – the encoder‑decoder architecture that underpins the OCR pipeline.
• PP‑DocLayout‑V3: A Fast and Accurate Layout Detector for Documents – the layout analysis component integrated into the two‑stage pipeline.

These papers collectively provide the theoretical foundation for multi‑modal token prediction, reinforcement‑learning‑based fine‑tuning, and efficient layout‑aware OCR.

Licensing Information

The repository’s README states a MIT license for the code and model weights, while the Hugging Face model card lists the license as “unknown”. In practice, the MIT licence is permissive:

• You may use, modify, distribute, and commercialise the software and model without paying royalties.
• Only the original copyright notice and licence text need to be retained in redistributed copies.
• No warranty is provided; you assume all risk of using the model in production.

If a downstream user wishes to embed GLM‑OCR in a commercial product, the MIT terms allow it, provided they keep the attribution. The “unknown” tag on the model card is likely a metadata oversight; the underlying repository’s LICENSE file clarifies the intent. Nevertheless, users should double‑check any third‑party data incorporated during fine‑tuning to ensure compliance with its own licences.