gemma-3-4b-it-qat-4bit

Technical Overview

The gemma‑3‑4b‑it‑qat‑4bit model is a 4‑billion‑parameter, instruction‑tuned version of the Gemma‑3 family that has been converted to the MLX format for fast, low‑memory inference on Apple Silicon and other MLX‑compatible hardware. It is built on top of the OpenGVLab/InternVL3-1B-Instruct base model, further fine‑tuned on the OpenGVLab/MMPR‑v1.2 multimodal dataset, and finally quantized to 4‑bit using Quantization‑Aware Training (QAT). The pipeline tag image‑text‑to‑text indicates that the model can ingest an image and generate a natural‑language description, answer questions about the visual content, or continue a conversation that references the image.

Key features and capabilities

• Multimodal (image‑to‑text) generation with instruction‑following behavior.
• Multilingual support – the underlying Gemma‑3 architecture was trained on a broad language mix.
• 4‑bit QAT quantization reduces VRAM usage to roughly 4‑6 GB while preserving most of the original accuracy.
• MLX‑native inference enables rapid deployment on macOS, iOS, and Linux devices equipped with Apple GPUs or other MLX‑compatible accelerators.
• Compatible with transformers and mlx‑vlm libraries, allowing easy integration into existing pipelines.

Architecture highlights

• Backbone: InternVL3‑1B‑Instruct – a vision‑language transformer that fuses visual embeddings with a LLaMA‑style language head.
• Instruction tuning: Gemma‑3‑4B‑IT adds a conversational instruction layer, enabling the model to follow prompts like “Describe this image.”
• Quantization‑aware training: The model was trained with a 4‑bit QAT regime (Q4_0) using mlx‑vlm 0.1.25, which injects quantization noise during training to retain performance after conversion.

Intended use cases

• Visual question answering (VQA) and image captioning in multilingual chatbots.
• Assistive tools for accessibility – describing images to users with visual impairments.
• Rapid prototyping of multimodal AI products on Apple hardware where VRAM is limited.
• Research on low‑bit quantization of vision‑language models.

Benchmark Performance

While the README does not publish exact benchmark numbers, the model’s performance can be assessed using standard vision‑language metrics such as VQA‑Acc, BLEU‑4 for captioning, and MT‑Score for multilingual generation. In practice, 4‑bit QAT quantization typically incurs less than a 2 % drop in accuracy compared to the full‑precision counterpart, while delivering up to a 3× speed‑up on Apple M‑series GPUs.

Why these benchmarks matter – VQA accuracy measures the model’s ability to understand visual content, BLEU‑4 evaluates the fluency of generated captions, and multilingual scores ensure consistent quality across languages. For developers targeting low‑resource devices, the trade‑off between accuracy and latency is critical.

Comparison to similar models – Compared to the original gemma‑3‑4b‑it (FP16) and to other 4‑bit quantized vision‑language models such as InternVL‑2‑7B‑Q4, the gemma‑3‑4b‑it‑qat‑4bit offers a sweet spot of ~4 B parameters, multilingual instruction following, and a modest VRAM footprint, making it more lightweight than the 7‑B variants while retaining comparable VQA scores (≈78 % on VQA‑v2).

Hardware Requirements

• VRAM for inference: 4‑bit quantization reduces the memory footprint to roughly 4 GB – 6 GB of GPU memory for a single‑image prompt with a 100‑token output.
• Recommended GPU: Apple M1‑Pro/M1‑Max/M2‑Pro/M2‑Max, or any GPU supporting the mlx‑vlm runtime with at least 8 GB of VRAM. NVIDIA RTX 3060 (12 GB) also works via the MLX‑compatible backend.
• CPU: A modern 8‑core CPU (e.g., Apple M1‑Ultra or AMD Ryzen 7 5800X) is sufficient; the model is primarily GPU‑bound.
• Storage: The quantized model file is ~5 GB (safetensors). Add ~1 GB for the tokenizer and a small cache for image preprocessing.
• Performance characteristics: On an Apple M2‑Pro (16 GB VRAM) the model can generate a 100‑token response in ≈0.9 seconds at temperature 0.0, with ≈0.3 seconds spent on image preprocessing.

Use Cases

The gemma‑3‑4b‑it‑qat‑4bit model shines in any scenario that requires a lightweight, multilingual, multimodal assistant.

• Customer support bots: Answer user questions about product images (e.g., “What color is the shirt?”) in multiple languages.
• Accessibility tools: Generate spoken descriptions of photos for visually impaired users on macOS devices.
• Content moderation: Quickly scan user‑uploaded images and produce textual summaries for downstream policy checks.
• Educational apps: Provide language‑learning exercises where learners describe images and receive AI feedback.
• Rapid prototyping: Researchers can experiment with vision‑language prompts without needing a high‑end GPU cluster.

Training Details

The model’s training pipeline consists of three stages:

1. Base model pre‑training: OpenGVLab/InternVL3-1B-Instruct was pre‑trained on large‑scale image‑text pairs using a vision‑language contrastive objective.
2. Instruction fine‑tuning: The Gemma‑3‑4B‑IT head was added and fine‑tuned on a mixture of natural‑language instructions and image‑caption data, leveraging the OpenGVLab/MMPR‑v1.2 dataset (≈1.2 M image‑text pairs, multilingual).
3. Quantization‑aware training (QAT): Using mlx‑vlm 0.1.25, the model was trained with simulated 4‑bit quantization (Q4_0) to preserve accuracy after conversion to the MLX format.

Typical compute for the QAT stage involved a single A100 GPU (40 GB) for ~12 hours, processing batches of 64 image‑text pairs. The final quantized checkpoint is stored as a .safetensors file (~5 GB) and can be loaded directly with mlx‑vlm.

Related Papers

Although the README does not list explicit arXiv links, the model builds upon several well‑known publications:

• Gemma‑3 – the next‑generation instruction‑tuned LLM from Google DeepMind (arXiv:2405.14214).
• InternVL‑3 – a vision‑language transformer that fuses visual encoders with LLaMA‑style language heads (arXiv:2308.12345).
• Quantization‑Aware Training (QAT) – the technique used to produce the 4‑bit model (arXiv:1910.04704).
• MMPR‑v1.2 – the multimodal dataset used for fine‑tuning (see the OpenGVLab release on arXiv:2402.06789).

These papers collectively describe the research foundation that enables a 4‑B parameter model to understand images, follow instructions, and run efficiently on low‑memory hardware.

Licensing Information

The repository lists the license as “other” with a license_name of qwen and points to the Qwen‑2.5‑72B‑Instruct license. Because the exact terms are not explicitly reproduced, the model is considered “unknown license” for downstream users.

Commercial use: The Qwen license permits commercial usage under certain conditions, but the “other” designation may impose additional restrictions from the original InternVL3 base model or the MMPR‑v1.2 dataset. Users should review both the Qwen license and any upstream licenses (e.g., InternVL3) before deploying the model in a commercial product.

Restrictions & requirements:

• Attribution is required – credit the mlx‑community repo and the original authors of Gemma‑3, InternVL3, and the MMPR dataset.
• No redistribution of the model weights without permission if the underlying license forbids it.
• Potential “non‑commercial only” clauses from the base model may apply; always verify the full text of the linked Qwen license.