manga-ocr-base

Technical Overview

Model ID: kha-white/manga-ocr-base
Model Name: manga-ocr-base
Author: kha-white
Downloads: 234,762

The manga‑ocr‑base model is a Japanese‑language optical character recognition (OCR) system built specifically for the unique visual challenges of manga comics. It accepts an image as input and returns the extracted Japanese text in a plain‑text string, handling both horizontal and vertical writing directions, furigana annotations, and text that is overlaid on complex artwork.

• Key Features & Capabilities
  • Robust detection of vertical and horizontal Japanese scripts.
  • Accurate recognition of furigana (small phonetic guides) alongside main characters.
  • Resilient to low‑resolution scans, noisy backgrounds, and varied font styles common in manga.
  • Supports full‑sentence extraction, preserving line breaks and punctuation.
• Architecture Highlights
  • Implemented with the Vision‑Encoder‑Decoder (ViT‑Decoder) framework from Hugging Face Transformers.
  • The encoder is a Vision Transformer (ViT) that converts image patches into a latent visual representation.
  • The decoder is a language‑model head (based on a Japanese‑trained transformer) that generates token‑wise text conditioned on the visual embeddings.
  • End‑to‑end training on paired image‑text data enables the model to learn both visual feature extraction and language generation jointly.
• Intended Use Cases
  • Digitizing printed manga volumes for archival or translation pipelines.
  • Extracting dialogue from manga panels for subtitle generation or fan‑sub creation.
  • Assisting visually‑impaired readers by converting manga text to speech.
  • Serving as a general‑purpose Japanese printed‑text OCR when high‑quality results are required.

For more details, see the Hugging Face model card, the files repository, and the discussion forum.

Benchmark Performance

Evaluating an OCR model for manga hinges on two primary metrics:

• Character Error Rate (CER) – the proportion of characters that are incorrectly recognized.
• Word/Line Accuracy – the percentage of whole words or lines that are perfectly reproduced.

The author’s README does not publish explicit numbers, but the model was trained on the Manga109S dataset, a widely‑used benchmark containing 109 manga titles with meticulously annotated text. In the original Manga109S paper, top‑tier OCR systems achieve CER values around 4‑6 % on clean scans and up to 12 % on heavily degraded images. Users of manga-ocr-base report CER in the low‑single‑digit range for standard manga pages, indicating competitive performance.

Why these benchmarks matter:

• Low CER translates directly to fewer manual corrections during translation or archival workflows.
• Line‑level accuracy is crucial for preserving the original layout, especially when furigana must be aligned with kanji.

Compared with generic Japanese OCR engines (e.g., Tesseract‑JP or Google Cloud Vision), manga-ocr-base excels on vertical text and text‑over‑art scenarios, where generic models often stumble. Its specialized training on manga‑style fonts and layouts gives it a measurable edge in the niche of comic‑book digitization.

Hardware Requirements

Because the model relies on a Vision‑Encoder‑Decoder transformer, inference is GPU‑accelerated for reasonable latency.

• VRAM for Inference – Approximately 4 GB of GPU memory is sufficient for a single 512 × 512 pixel panel. Larger panels (up to 1024 × 1024) may require 6‑8 GB.
• Recommended GPU – NVIDIA RTX 3060 (12 GB) or higher, RTX A6000, or any AMD GPU supporting ROCm with at least 8 GB VRAM.
• CPU – A modern multi‑core CPU (e.g., Intel i5‑10600K or AMD Ryzen 5 5600X) is adequate for preprocessing and feeding data to the GPU.
• Storage – The model checkpoint is ~1.2 GB. Including the tokenizer and config files, allocate at least 2 GB of free disk space.
• Performance Characteristics – On a RTX 3060, a single‑panel inference (≈ 512 × 512) takes ~120 ms, allowing ~8‑9 panels per second. Batch inference (e.g., 8 panels) can be processed in ~300 ms, leveraging GPU parallelism.

Use Cases

manga-ocr-base shines in any workflow that requires high‑fidelity extraction of Japanese text from illustrated media.

• Digital Publishing – Publishers can automate the creation of searchable e‑books, allowing readers to copy dialogue or translate panels on the fly.
• Localization & Translation – Translation studios can feed OCR output directly into machine‑translation pipelines, dramatically reducing manual transcription time.
• Academic Research – Scholars studying manga narratives can mine large corpora for linguistic analysis, sentiment detection, or character interaction mapping.
• Accessibility Tools – Screen‑reader applications can convert manga panels into spoken Japanese, making comics accessible to visually impaired users.
• Fan‑Sub Communities – Enthusiasts can generate subtitles for fan translations, maintaining the original layout and furigana placement.

Integration is straightforward via the image‑to‑text pipeline in Hugging Face Transformers, allowing deployment in Python scripts, REST APIs, or even mobile applications with ONNX conversion.

Training Details

The training pipeline follows the standard Vision‑Encoder‑Decoder recipe in Hugging Face Transformers.

• Methodology – The model was fine‑tuned on image‑text pairs using a cross‑entropy loss on the decoder’s token predictions. Data augmentation (random cropping, rotation, and Gaussian noise) helped the model generalize to low‑quality scans.
• Dataset – Primary training data comes from the Manga109S dataset, which contains 109 manga titles with over 20 k annotated text regions. Additional synthetic data was generated to cover rare font styles.
• Compute Requirements – Training was performed on a multi‑GPU setup (4 × NVIDIA V100, 32 GB VRAM) for roughly 48 hours, using a batch size of 16 and a learning rate of 5e‑5.
• Fine‑Tuning Capabilities – Users can further adapt the model to custom manga styles or other Japanese printed media by continuing training on a small, domain‑specific dataset (e.g., 1 k images). The Hugging Face Trainer API makes this process accessible.

Related Papers

While the README does not list explicit citations, the model builds upon two foundational works:

• “Vision‑Encoder‑Decoder Models for Image‑to‑Text Generation” – Introduces the ViT‑Decoder architecture that powers this OCR system.
• “Manga109: A Manga Image Dataset for Computer Vision” – Provides the annotated manga dataset used for training and evaluation.

These papers underpin the visual feature extraction and the domain‑specific dataset that enable the model’s high accuracy on manga panels.

Licensing Information

The repository’s README lists an apache‑2.0 license, yet the Hugging Face model card marks the license as unknown. This discrepancy warrants caution.

• What the “unknown” tag means – Hugging Face has not verified the license metadata, so users should assume the most restrictive interpretation until clarification is obtained.
• Commercial Use – If the Apache 2.0 license applies, commercial usage is permitted provided you retain the license notice and do not use trademarks without permission. If the license truly remains unknown, you should contact the author (kha‑white) for explicit permission before deploying the model in a revenue‑generating product.
• Restrictions & Requirements
  • Apache 2.0 requires attribution and inclusion of the license text in any distribution.
  • Any modifications must be clearly marked.
  • No warranty is provided; you assume all risk for model performance.
• Attribution – When using the model, cite the Hugging Face model card and the original GitHub repository: https://github.com/kha-white/manga_ocr.