GLM-4.5-Air-AWQ-4bit

Technical Overview

The GLM‑4.5‑Air‑AWQ‑4bit model (model ID cyankiwi/GLM-4.5-Air-AWQ-4bit) is a 4‑bit quantized variant of the GLM‑4.5‑Air foundation model released by Zhipu AI. It is built on the zai‑org/GLM‑4.5‑Air base and targets high‑throughput text‑generation and conversational AI workloads while dramatically reducing memory footprint. The model supports both English and Chinese (and additional multilingual extensions) and is optimized for the text‑generation pipeline in the 🤗 Transformers ecosystem.

Key capabilities include:

• Hybrid reasoning with “thinking” and “non‑thinking” modes, enabling complex tool‑use and chain‑of‑thought reasoning.
• 106 B total parameters (12 B active) – a compact yet powerful version of the 355 B GLM‑4.5 family.
• 4‑bit integer quantization (AWQ v1.0, group‑size 32) that cuts VRAM usage from ~206 GB to ~59 GB.
• Compatibility with vLLM, SGLang and the standard 🤗 Transformers API.

Architecturally, GLM‑4.5‑Air follows a mixture‑of‑experts (MoE) design (named glm4_moe in Transformers) where a subset of expert feed‑forward layers is activated per token, yielding a high effective parameter count while keeping inference cost low. The AWQ quantizer preserves the original weight distribution by applying per‑group scaling factors, which explains the minimal perplexity increase (from 1.3855 to 1.4032) after quantization.

Intended use cases revolve around intelligent agents, chat assistants, code generation, and any application that benefits from strong reasoning while operating under limited GPU memory budgets.

Benchmark Performance

GLM‑4.5‑Air‑AWQ‑4bit has been evaluated on the same 16 K‑token context length as its FP16 counterpart. The primary metric reported is perplexity:

• GLM‑4.5‑Air (FP16): 1.38554
• GLM‑4.5‑Air‑AWQ‑4bit (INT4): 1.40319

A perplexity rise of only ~0.017 demonstrates that the 4‑bit AWQ quantization preserves language modeling quality while cutting memory usage by ~71 %. This benchmark is crucial for generative LLMs because lower perplexity correlates with more fluent and accurate text generation. Compared with other open‑source 4‑bit models (e.g., LLaMA‑2‑13B‑Q4, Mistral‑7B‑INstruct‑Q4), GLM‑4.5‑Air‑AWQ‑4bit offers a superior trade‑off: a larger active parameter count (12 B) and competitive perplexity, positioning it among the top‑performing quantized MoE models on the market.

Hardware Requirements

Running the model at full 16 K context length requires roughly 59 GB of VRAM. This is the memory footprint after AWQ quantization; the original FP16 model would need ~206 GB.

• GPU recommendation: NVIDIA A100 40 GB (dual‑GPU) or RTX 4090 24 GB with tensor‑parallelism = 2 (requires --enable-expert-parallel).
• CPU: Any modern x86‑64 CPU; for optimal throughput, a 12‑core Intel Xeon or AMD EPYC with high‑speed RAM (≥ 256 GB) is advised.
• Storage: Model files (safetensors + tokenizer) total ~60 GB; SSD/NVMe storage is recommended for fast loading.
• Inference speed: Using vLLM, a single A100 can serve ~30 tokens/s for 16 K context; scaling to multi‑GPU with --tensor-parallel-size improves throughput but requires the --enable-expert-parallel flag.

Use Cases

GLM‑4.5‑Air‑AWQ‑4bit shines in scenarios where strong reasoning meets limited GPU memory:

• Intelligent conversational agents: Customer‑service bots that need to reason over long histories (up to 16 K tokens).
• Code assistance: Autocompletion or debugging tools that benefit from the model’s hybrid reasoning and tool‑use capabilities.
• Multilingual content creation: English‑Chinese mixed‑language generation for marketing, translation, or social media.
• Research prototyping: Researchers can fine‑tune the 4‑bit checkpoint on domain‑specific data while staying within a single‑GPU budget.

Training Details

While the README does not disclose the full training pipeline, the following information is known:

• Base model: zai‑org/GLM‑4.5‑Air – a 106 B parameter MoE model trained on a mixture of multilingual internet text, code, and instruction data.
• Quantization: Post‑training AWQ v1.0 (4‑bit, group‑size 32) using the llm‑compressor tool and calibrated on the NVIDIA Llama‑Nemotron post‑training dataset.
• Fine‑tuning: The quantized checkpoint can be further fine‑tuned with LoRA or QLoRA in the 🤗 Transformers ecosystem, as the underlying architecture remains unchanged.
• Compute: Training the original FP16 model required multi‑node GPU clusters (A100 80 GB or equivalent). The quantization step itself is lightweight and can be performed on a single 40 GB GPU.

Related Papers

The model’s scientific foundation is described in the following open‑access works:

• GLM‑4.5 Technical Report (arXiv:2508.06471) – introduces the GLM‑4.5 family, architecture, and benchmark results.
• Additional references are linked from the Zhipu AI technical blog and documentation (see the model card for more URLs).

Licensing Information

The model card lists the license field as MIT, while the tag section includes license: unknown. The MIT license is permissive: you may use, modify, distribute, and even commercialize the model without paying royalties, provided you retain the original copyright notice and license text.

• Commercial use: Allowed under MIT; you can embed the model in SaaS products, chatbots, or any revenue‑generating service.
• Restrictions: None beyond the standard MIT attribution requirement. No patent or trademark claims are indicated.
• Attribution: Include a copy of the MIT license and credit “cyankiwi / Zhipu AI” when redistributing.
• Unknown tag note: Some platforms may flag the model as “unknown license”; double‑check the model card for the explicit MIT statement before commercial deployment.