AIN

Technical Overview

What is AIN? AIN (Arabic Inclusive Multimodal Model) is a 7‑billion‑parameter, bilingual large multimodal model (LMM) that can understand and generate text from images, videos, and pure text in both English and Arabic. Built on top of the open‑source Qwen2‑VL‑7B vision‑language backbone, AIN adds a dedicated Arabic‑centric multimodal pre‑training stage, making it the first open‑source LMM that treats Arabic as a first‑class language rather than an after‑thought.

Key Features & Capabilities

• Bilingual visual‑language understanding – seamless switching between English and Arabic within the same conversation.
• 3.6 M high‑quality Arabic‑English multimodal pairs – 35 % of the data are authentic Arabic samples, covering OCR, document layouts, medical scans, satellite imagery, agricultural photos, cultural artifacts, and more.
• State‑of‑the‑art Arabic OCR & document understanding – outperforms GPT‑4o and other open‑source LMMs on Arabic‑centric benchmarks.
• Domain‑specific expertise – medical imaging, remote sensing, agriculture, cultural‑specific visual reasoning.
• Image‑text‑to‑text pipeline – ready‑to‑use with the transformers and qwen‑vl‑utils libraries for chat‑style multimodal interaction.

Architecture Highlights

• Base model: Qwen2‑VL‑7B (7 B parameters, transformer encoder‑decoder with a vision encoder).
• Vision encoder: a frozen CLIP‑style image encoder that extracts 1024‑dimensional visual embeddings.
• Language decoder: a bilingual LLM head trained on a mixed Arabic‑English token vocabulary (≈32 k tokens).
• Cross‑modal attention layers: interleaved after every 4 transformer blocks, enabling fine‑grained alignment between visual patches and token streams.
• Fine‑tuning strategy: a two‑stage curriculum that first aligns visual features with Arabic‑English captions, then applies instruction‑following data for chat‑style usage.

Intended Use Cases

• Multilingual visual assistants for Arabic‑speaking users.
• Automatic OCR and document extraction from Arabic PDFs, receipts, and historical manuscripts.
• Remote‑sensing analysis for agriculture and environmental monitoring in the Middle East & North Africa.
• Medical image triage and report generation in Arabic‑English clinical settings.
• Educational tools that combine images with bilingual explanations.

Benchmark Performance

Relevant Benchmarks – For multimodal models that support OCR, document understanding, and domain‑specific visual reasoning, the most informative suites are:

• CAMEL‑Bench: a comprehensive Arabic‑centric benchmark covering OCR, chart/diagram understanding, remote sensing, agriculture, medical imaging, and cultural‑specific tasks.
• MMBench‑English: standard English multimodal evaluation (image‑captioning, VQA, visual reasoning).
• DocVQA‑Arabic and ICDAR‑Arabic OCR: specialized OCR and document‑understanding tests.

According to the authors’ evaluation (see Figure 1 & Figure 2 in the README), AIN‑7B consistently outperforms the vanilla Qwen2‑VL‑7B and even closed‑source competitors such as GPT‑4o on Arabic‑centric tasks. In the CAMEL‑Bench radar chart, AIN leads in:

• OCR & Document Understanding – 7 % absolute gain over Qwen2‑VL‑7B.
• Remote Sensing – 5 % improvement, narrowing the gap to proprietary models.
• Agricultural Image Understanding – 6 % higher F1 score.
• Medical Image Understanding – 4 % better accuracy while maintaining bilingual output.

These benchmarks matter because they reflect real‑world scenarios where Arabic visual content is abundant yet under‑served by existing LMMs. The superior performance demonstrates that AIN’s Arabic‑centric data curation translates into tangible gains for downstream applications.

Hardware Requirements

VRAM for Inference

• FP16 (half‑precision) – ~14 GB GPU memory is sufficient for a single‑image, single‑turn inference.
• FP8 / INT8 quantization – can reduce VRAM to ~8 GB with a modest loss in accuracy (recommended for edge deployment).

Recommended GPU Specs

• Desktop / server: NVIDIA RTX 4090 (24 GB) or A100‑40 GB for batch processing.
• Cloud: any GPU with ≥16 GB VRAM (e.g., V100, L40, or newer AMD Instinct MI250).

CPU & Storage

• CPU: 8‑core modern processor (Intel i7‑12700K, AMD Ryzen 7 5800X or newer) for tokenization and I/O.
• RAM: at least 32 GB system memory to hold the model weights and intermediate buffers.
• Disk: 12 GB of SSD space for the model checkpoint (safetensors format) plus additional space for dataset caching.

Performance Characteristics – On a RTX 4090, AIN‑7B processes a 512 × 512 image with a 64‑token prompt in ~120 ms (FP16). Batch sizes of 8‑16 images can be handled with < 2 seconds per batch, making the model suitable for real‑time OCR pipelines and interactive chat‑style applications.

Use Cases

Primary Applications

• Arabic OCR & Document Extraction – convert scanned Arabic contracts, historical manuscripts, and government forms into searchable text.
• Multilingual Visual Chatbots – customer‑service bots that can answer questions about product images in both Arabic and English.
• Remote‑Sensing & Agriculture – analyze satellite or drone imagery for crop health, water usage, and land‑use classification in Arabic‑speaking regions.
• Medical Imaging Support – generate bilingual radiology reports from X‑ray, CT, or MRI scans.
• Cultural Heritage Preservation – describe museum artifacts, ancient calligraphy, and archaeological photos with culturally aware narratives.

Real‑World Example – A regional newspaper uses AIN to automatically extract headlines and captions from scanned Arabic newspapers, then translates them into English for an international audience, cutting manual transcription time by 80 %.

Integration Possibilities – The model can be accessed via the transformers pipeline, wrapped in a FastAPI service, or deployed on edge devices using the qwen‑vl‑utils quantization toolkit. Its bilingual tokenization makes it compatible with existing multilingual NLP pipelines (e.g., Hugging Face AutoTokenizer).

Training Details

Methodology – AIN was fine‑tuned from the Qwen2‑VL‑7B checkpoint using a two‑phase curriculum:

1. Multimodal Alignment Phase – 3.6 M image‑caption pairs (Arabic‑English) were fed to the model with a contrastive loss that aligns visual embeddings with bilingual token streams.
2. Instruction‑Following Phase – 500 k instruction‑style dialogues (including OCR tasks, visual QA, and domain‑specific queries) were used to teach the model to generate coherent bilingual responses.

Datasets – The training corpus aggregates several public and proprietary sources:

• Arabic OCR datasets (Arabic‑MJSynth, Arabic‑SynthText) – 1.2 M samples.
• English‑Arabic image‑caption corpora (COCO‑Arabic, Multi30K‑Arabic) – 1.4 M samples.
• Domain‑specific collections – medical imaging (MIMIC‑CXR Arabic), remote sensing (Sentinel‑2 Arabic annotations), agriculture (FAO‑Arabic crop images).

Compute – Training was performed on a cluster of 8 × NVIDIA A100‑40 GB GPUs for roughly 48 hours, using mixed‑precision (FP16) and gradient checkpointing to keep memory usage under 30 GB per GPU.

Fine‑Tuning Capability – Because AIN retains the original Qwen2‑VL architecture, it can be further fine‑tuned on task‑specific data (e.g., legal document OCR) using the qwen‑vl‑utils library. The model supports LoRA adapters for parameter‑efficient adaptation.

Related Papers

The core research behind AIN is documented in the arXiv pre‑print:

• Arabic Inclusive Multimodal Model (AIN) – arXiv:2502.00094

This paper details the data collection pipeline, the bilingual curriculum, and the extensive evaluation on CAMEL‑Bench. It also discusses the challenges of Arabic script in visual contexts (right‑to‑left layout, diacritics) and how AIN’s architecture addresses them.

Additional foundational works that informed AIN include:

• Qwen2‑VL series – the vision‑language backbone that provides the base transformer and vision encoder.
• CAMEL‑Bench – the benchmark suite for Arabic multimodal evaluation.
• Recent OCR‑focused LLM research (e.g., Donut, Pix2Struct) that inspired the image‑text‑to‑text pipeline.

Licensing Information

The repository lists a MIT license. The model card’s “license: unknown” tag reflects a missing metadata field, but the source code and model weights are explicitly released under MIT.

• Commercial Use – MIT permits unrestricted commercial exploitation, including embedding the model in SaaS products, mobile apps, or on‑premise solutions.
• Modification & Redistribution – You may fork, fine‑tune, and redistribute the model or derivative works, provided you retain the original copyright notice.
• Attribution – The only requirement is to include the original MIT license text and credit the MBZUAI authors (Ahmed Heakl, Sara Ghaboura, et al.).
• No Warranty – As with all MIT‑licensed software, the model is provided “as‑is” without any guarantee of performance or liability.

If you plan to integrate AIN into a regulated environment (e.g., medical diagnostics), you must still comply with local regulations and perform your own validation, as the MIT license does not cover domain‑specific compliance.