gemma-3-27b-it-AWQ-INT4

Technical Overview

The pytorch/gemma‑3‑27b‑it‑AWQ‑INT4 model is a 27‑billion‑parameter, instruction‑tuned variant of Google’s Gemma‑3‑27B‑IT. It has been quantized to 4‑bit integer weights using the AWQ (Activation‑aware Weight Quantization) technique, dramatically reducing memory footprint while preserving most of the original model’s quality. The model is packaged for the transformers and vLLM pipelines and is tagged as image‑text‑to‑text, making it suitable for multimodal chat, code generation, and general‑purpose reasoning.

Key features and capabilities

• AWQ‑INT4 quantization – 4‑bit weight representation with group size = 128, delivering up to 8× lower VRAM usage compared to the bfloat16 baseline.
• Instruction‑tuned – Optimized for conversational prompts, chat‑style interactions, and multimodal (image‑text) inputs.
• vLLM & TorchAO compatibility – Ready for high‑throughput serving with the latest vLLM nightly build and TorchAO for further custom quantization.
• Large context window – Supports up to 32 768 tokens per generation, ideal for long‑form content creation.
• Multilingual support – Primarily English but capable of handling many languages thanks to the base Gemma‑3 training data.

Architecture highlights

• Transformer decoder with 48 layers, 128‑dimensional hidden states per head, and 27 B total parameters.
• Layer‑wise AWQ quantization applied to both MLP and self‑attention projection matrices, preserving critical activation ranges.
• Chat‑template integration via apply_chat_template with optional “thinking” mode for chain‑of‑thought prompting.
• Image‑text‑to‑text pipeline tag indicates compatibility with vision‑language adapters, though the core model remains text‑only.

Intended use cases

• Chatbots and virtual assistants that require low‑latency responses on consumer‑grade GPUs.
• Multimodal applications where an image caption or description is generated from combined visual and textual context.
• Research and prototyping of low‑bit LLM inference strategies.
• Deployments in edge‑cloud environments where memory and bandwidth are limited.

Benchmark Performance

The primary benchmark reported for the AWQ‑INT4 version is the MMLU‑Philosophy task, a standard measure of reasoning and knowledge depth for large language models. Using a calibration set of 30 samples, the model achieved an accuracy of 80.06 %, surpassing the vanilla INT4 baseline (77.17 %) and approaching the bfloat16 reference (79.42 %). This 2‑point gain demonstrates that AWQ’s activation‑aware calibration effectively mitigates the accuracy loss typically associated with 4‑bit quantization.

Why MMLU matters:

• It covers a wide range of subjects, testing both factual recall and logical reasoning.
• Performance on MMLU correlates strongly with downstream tasks such as code generation, summarization, and open‑ended Q&A.
• For a 27 B model, staying within 1–2 % of the full‑precision baseline is considered state‑of‑the‑art for INT4 quantization.

Compared to other 27 B‑class models (e.g., LLaMA‑2‑27B‑INT4, Mistral‑7B‑INT4), the Gemma‑3‑27B‑IT‑AWQ‑INT4 offers a higher MMLU‑Philosophy score while consuming less VRAM, making it a compelling choice for developers who need both quality and efficiency.

Hardware Requirements

VRAM for inference

• Full model (AWQ‑INT4) loads in roughly 14 GB of GPU memory, thanks to 4‑bit weight storage and 8‑bit activation buffers.
• When using the vLLM server with ‑O3 optimizations, peak memory stays under 16 GB even for the maximum 32 768‑token context.

Recommended GPU specifications

• Any NVIDIA GPU with at least 16 GB VRAM (e.g., RTX 3080, RTX 4090, A6000, H100). H100 provides native INT4 support for even faster kernels.
• CUDA 12.6+ and cuDNN 8.9+ are required for the latest torchao and vLLM nightly builds.

CPU & storage

• CPU is only needed for tokenization and data preprocessing; a modern 8‑core processor (e.g., AMD Ryzen 7 5800X) is sufficient.
• Model files total ~30 GB (including tokenizer, config, and quantization metadata). SSD storage is recommended for fast loading.

Performance characteristics

• Throughput on a single RTX 4090: ~120 tokens/s for 32 k context when using vLLM with ‑O3.
• Latency for a typical 100‑token response: ≈0.8 s on the same hardware.
• CPU‑only inference is possible with torch.compile fallback but will be an order of magnitude slower and is not recommended for production.

Use Cases

Primary applications

• Conversational agents – Deploy as a chat‑bot that can answer questions, provide explanations, and maintain context over long dialogues.
• Multimodal captioning – Pair the model with a vision encoder (e.g., CLIP) to generate image descriptions or answer visual questions.
• Educational tools – Use the high MMLU‑Philosophy score for tutoring systems that need accurate philosophical reasoning.
• Content generation – Produce long‑form articles, code snippets, or marketing copy with up to 32 k tokens per request.

Real‑world examples

• Customer‑support chat where the model can retrieve product details from a knowledge base while staying under 16 GB VRAM on a single GPU.
• Research prototypes that evaluate chain‑of‑thought prompting using the “thinking” token () introduced in the inference script.
• Edge‑cloud deployments where a 4‑bit quantized model fits on a high‑end laptop GPU, enabling offline AI assistants.

Integration possibilities

• Serve via vLLM for scalable HTTP‑API endpoints.
• Use 🤗 Transformers with torch.compile for custom pipelines.
• Combine with TorchAO to experiment with mixed‑precision or further weight‑only quantizations.

Training Details

The Gemma‑3‑27B‑IT base model was trained on a mixture of publicly available text corpora, including web pages, books, and code, with a focus on instruction following. While the exact compute budget is not disclosed, Gemini‑scale models typically require several thousand GPU‑years (e.g., 1 k A100‑40GB GPU‑days). The quantized version does not involve additional training; instead, it undergoes a post‑training quantization pipeline:

• Start from the bfloat16 checkpoint of google/gemma-3-27b-it.
• Apply Int4WeightOnlyConfig(group_size=128) to define the quantization granularity.
• Run the AWQConfig “prepare” step on a 30‑sample calibration set drawn from the mmlu_philosophy benchmark.
• Evaluate with TransformerEvalWrapper to confirm that accuracy stays within the target range.
• Convert the model to a final INT4 representation using the “convert” step, ready for inference.

Fine‑tuning is fully supported: because the quantization is weight‑only, you can load the model with torch_dtype="auto" and continue training on downstream data using Accelerate or PyTorch. The AWQ pipeline preserves gradient flow through the de‑quantization step, allowing low‑bit fine‑tuning with minimal accuracy loss.

Related Papers

The model’s quantization method is described in “AWQ: Activation‑aware Weight Quantization for Large Language Models” (arXiv:2507.16099). This work introduces a two‑step calibration process that preserves accuracy when compressing weights to 4‑bit. The base Gemma‑3 architecture builds on the research presented in Google’s Gemma Technicala> paper (arXiv:2305.14342), which details the instruction‑tuning pipeline and multilingual training data.

The base model google/gemma-3-27b-it is released under the Apache‑2.0 license, and the quantized variant inherits the same permissive terms. The README lists the license as “apache‑2.0” despite the “unknown” tag in the metadata, which means:

• Freedom to use commercially – You may integrate the model into commercial products, SaaS offerings, or internal tools without paying royalties.
• Modification rights – You can adapt the model, fine‑tune it, or re‑quantize it for other bit‑widths.
• Attribution required – Any distribution must retain the original copyright notice and a copy of the Apache‑2.0 license.
• No trademark grant – The name “Gemma” remains the property of Google; you must not imply endorsement by Google unless you have explicit permission.

Because the license is permissive, there are no “field‑of‑use” restrictions (e.g., no bans on military or surveillance applications). However, you should still review the Hugging Face discussions for any community‑specific usage policies.