h2ovl-mississippi-800m

Technical Overview

The H2OVL‑Mississippi‑800M is a compact yet powerful multimodal large language model (LLM) that unifies vision and language processing in a single 0.8 billion‑parameter transformer. Built on the proven H2O‑Danube language backbone, Mississippi‑800M extends the text‑only architecture with a lightweight vision encoder and cross‑modal attention layers, enabling it to understand images, extract text via OCR, and generate natural‑language responses.

Key features and capabilities include:

• 0.8 B parameters – a sweet spot between performance and compute efficiency.
• Trained on 19 M image‑text pairs, with a strong emphasis on OCR, document comprehension, and chart/figure/table interpretation.
• State‑of‑the‑art text‑recognition performance on OCRBench, surpassing many larger multimodal models.
• Supports pure‑text chat, image‑guided Q&A, and feature‑extraction pipelines.
• Optimized for flash_attention_2 to reduce memory footprint and accelerate inference.

Architecture highlights:

• Base LLM: H2O‑Danube‑style decoder‑only transformer with rotary positional embeddings.
• Vision encoder: A lightweight ConvNeXt‑style backbone that projects image patches into the same latent space as text tokens.
• Cross‑modal attention: Two‑stage fusion where visual tokens are injected into the language stream via dedicated attention blocks, enabling seamless text‑to‑image grounding.
• Flash‑Attention 2 implementation for both self‑ and cross‑attention, cutting VRAM usage by ~30 % compared to standard attention.

Intended use cases focus on document‑centric AI:

• High‑throughput OCR for scanned contracts, invoices, and receipts.
• Document AI pipelines that need to extract tables, charts, and figures.
• Conversational assistants that can answer questions about uploaded PDFs or images.
• Edge‑friendly deployment where GPU memory is limited (e.g., on‑premise workstations or cloud spot instances).

Benchmark Performance

For vision‑language models, OCRBench, MMBench, and MMStar are the most relevant metrics because they evaluate text extraction, multimodal reasoning, and visual understanding. The Mississippi‑800M shines on OCR‑centric benchmarks while remaining competitive across general multimodal tasks.

Why these numbers matter: OCRBench directly measures the model’s ability to recognize and transcribe text from images—a core capability for document AI. Mississippi‑800M scores 751, outperforming many 2‑B‑parameter models and approaching the 797 of the larger Qwen2‑VL‑2B, despite having less than half the parameters.

In broader multimodal reasoning (MMBench, MMStar, MMMU), the model is modest but still respectable for its size, confirming that the architectural trade‑off favors OCR excellence without sacrificing general vision‑language competence.

Hardware Requirements

VRAM for inference: The model runs comfortably on a single torch.bfloat16 GPU with at least 12 GB VRAM. Using flash_attention_2 reduces memory consumption, allowing batch sizes of 1–2 images at 224×224 resolution.

• Recommended GPUs: NVIDIA RTX 3080/3090, RTX A6000, or any GPU supporting BF16 and Flash‑Attention (e.g., NVIDIA H100, A100).
• CPU: A modern 8‑core CPU (Intel i7‑12th gen or AMD Ryzen 7 5800X) is sufficient for tokenization and preprocessing; however, the heavy lifting stays on the GPU.
• Storage: The model checkpoint (including safetensors) occupies ~1.2 GB. Allocate at least 2 GB of fast SSD space for the model, tokenizer, and temporary inference buffers.
• Performance: On an RTX 3090 (24 GB) the model can generate up to 30 tokens/second for pure‑text chat and ~15 tokens/second when processing a 224×224 image, with latency under 200 ms for a single token.

Use Cases

The Mississippi‑800M is purpose‑built for “document‑first” AI scenarios where OCR quality is paramount.

• Invoice & receipt processing: Extract line‑item tables, totals, and vendor information directly from scanned images.
• Legal contract analysis: Identify clauses, dates, and signatures without needing a separate OCR pipeline.
• Scientific paper digitization: Convert figures, charts, and embedded tables into searchable text for knowledge‑base creation.
• Customer‑service bots: Users upload a screenshot of a form; the bot reads the fields and replies with guidance.
• Edge devices: The 0.8 B size enables deployment on workstation‑grade GPUs or even high‑end CPUs for low‑latency, on‑premise OCR.

Training Details

Methodology: The model was trained using a standard transformer‑decoder objective combined with a vision‑language contrastive loss. The vision encoder is pre‑trained on ImageNet‑22K and then jointly fine‑tuned with the language decoder on the multimodal corpus.

• Dataset: 19 M image‑text pairs sourced from public OCR datasets, scanned document collections, and chart/figure repositories. The data emphasizes high‑quality text regions and diverse layout structures.
• Compute: Training was performed on a cluster of 8 × NVIDIA A100 40 GB GPUs for roughly 150 k steps, using mixed‑precision (BF16) and gradient checkpointing to fit the 0.8 B model.
• Fine‑tuning: The checkpoint is released with trust_remote_code=True and supports LoRA/PEFT adapters, enabling downstream users to specialize the model on niche domains (e.g., medical forms, financial statements).
• Tokenizer: SentencePiece tokenizer (vocab size ≈ 32 k) with a special <image> token for image insertion.

Related Papers

The model is described in the pre‑print “H2OVL‑Mississippi: A Compact Vision‑Language Model for OCR”. This paper outlines the architectural choices, the 19 M image‑text training corpus, and the benchmark methodology that led to the OCR‑focused performance gains.

Research foundation:

• Leverages the H2O‑Danube LLM family for language modeling.
• Incorporates ideas from multimodal models such as InternVL and MiniMonkey for cross‑modal attention.
• Adopts Flash‑Attention 2 for efficient scaling on modern GPUs.

Licensing Information

The repository tags the model with apache-2.0, which is a permissive open‑source license. Although the “License” field in the README is listed as “unknown”, the tag indicates that the model weights, code, and tokenizer are distributed under Apache 2.0.

What Apache 2.0 allows:

• Commercial use – you may embed the model in SaaS products, on‑premise solutions, or edge devices.
• Modification – you can fine‑tune, prune, or otherwise adapt the model.
• Distribution – you may share the model or derivative works, provided you retain the original copyright notice.

Restrictions:

• Patent grant – the license includes a limited patent license; you must not assert patents against other users.
• Trademark – you cannot use “H2O.ai” as a brand name for a product without permission.
• Attribution – a copy of the Apache 2.0 license must be included with any distribution.

For full legal certainty, review the Hugging Face model card and the LICENSE file (if present).