GLM-4.7-Flash-AWQ-4bit

Technical Overview

GLM‑4.7‑Flash‑AWQ‑4bit is a 4‑bit quantized version of the GLM‑4.7‑Flash large language model (LLM) released by the Z‑AI research team. The base model is a 30‑billion‑parameter Mixture‑of‑Experts (MoE) architecture that balances the performance of a 30 B dense model with the efficiency of a 3 B dense model. By applying the llm‑compressor AWQ v1.0 quantizer (group size = 32) to the original checkpoint, the model’s memory footprint shrinks from ~58 GB to ~17 GB while incurring only a 0.42 % increase in perplexity.

Key capabilities include:

• Multi‑language generation (English, Chinese, Arabic, etc.)
• Conversational and instruction‑following behavior
• Tool‑calling support via the glm47 and glm45 parsers
• Preserved “Thinking Mode” for complex multi‑turn tasks (τ²‑Bench, Terminal Bench)

Architecture highlights:

• MoE design with 3 B active parameters per token, 30 B total parameters
• Flash‑optimized transformer blocks (kernel‑level speedups for modern GPUs)
• AWQ 4‑bit quantization with 32‑wide groups, calibrated on the NVIDIA Nemotron Post‑Training Dataset v2

Intended use cases focus on high‑throughput text generation, chat‑bots, code assistance, and any scenario where a 30 B‑class LLM is desirable but hardware resources are limited. The model is fully compatible with the text‑generation pipeline in 🤗 Transformers and can be served via vLLM or SGLang.

Benchmark Performance

Benchmarking for GLM‑4.7‑Flash‑AWQ‑4bit focuses on two primary aspects: perplexity (a language‑model intrinsic metric) and task‑specific scores on widely‑used evaluation suites. The README reports a perplexity of 1.6096 versus 1.60292 for the full‑precision model—a negligible 0.417 % loss.

Task‑level results (selected from the original GLM‑4.7‑Flash paper) illustrate competitive performance against other 30 B‑class models:

These benchmarks matter because they test reasoning, code generation, browsing, and tool‑use abilities—core capabilities for modern LLM‑powered assistants. The AWQ‑4bit version retains most of the original model’s strength while enabling deployment on a single GPU with < 20 GB VRAM, a decisive advantage for startups and research labs with limited hardware budgets.

Hardware Requirements

The quantized checkpoint reduces the VRAM demand dramatically. According to the memory‑usage table, GLM‑4.7‑Flash‑AWQ‑4bit occupies ≈ 17.2 GB of GPU memory for a single‑GPU inference session (tensor‑parallel size < 8). For optimal performance, the following hardware profile is recommended:

• GPU: NVIDIA A100 (40 GB) or RTX 4090 (24 GB) – both comfortably fit the model and allow room for KV‑cache during long context windows (up to 131 072 tokens).
• CPU: Modern x86‑64 with ≥ 8 cores; the inference pipeline (vLLM/SGLang) benefits from fast NVMe‑backed storage and a decent CPU cache for token‑level preprocessing.
• RAM: Minimum 32 GB system memory; 64 GB+ is advisable when running multiple concurrent requests or large batch sizes.
• Storage: The model files (safetensors) total roughly 12 GB. SSD/NVMe storage ensures rapid model loading and low latency for the first‑run warm‑up.
• Network: If serving over a network, a 10 Gbps Ethernet connection reduces request latency for high‑throughput APIs.

Performance notes:

• Inference speed scales with the --tensor-parallel-size flag; keep it below 8 as per the “Known Issues”.
• KV‑cache for a 131 072‑token context consumes ~4 GB additional VRAM; plan head margin.
• vLLM’s pre‑compiled kernels (enabled via VLLM_USE_PRECOMPILED=1) provide the best throughput on the supported GPUs.

Use Cases

GLM‑4.7‑Flash‑AWQ‑4bit shines in scenarios where high‑quality text generation is needed but hardware budgets are constrained. Typical applications include:

• Chatbots & virtual assistants: Multi‑turn conversational agents that can call external tools (search, code execution) via the built‑in parsers.
• Code generation & debugging: The model’s strong performance on SWE‑bench Verified makes it suitable for IDE‑integrated coding assistants.
• Knowledge‑base Q&A: Large context windows (up to 131 k tokens) enable retrieval‑augmented generation for enterprise documents.
• Multilingual content creation: Supports English, Chinese, Arabic and other languages out‑of‑the‑box, useful for global marketing copy.
• Research prototyping: Low VRAM footprint allows academic labs to experiment with MoE‑style LLMs without a multi‑GPU cluster.

Integration pathways:

• Deploy via vLLM for high‑throughput API services.
• Wrap in 🤗 Transformers pipelines for quick prototyping in Python.
• Use the --tool-call-parser glm47 flag to enable structured tool calls in custom agents.

Training Details

While the exact training pipeline for the quantized checkpoint is not disclosed, the base model (GLM‑4.7‑Flash) follows the methodology outlined in the GLM‑4.5 technical report:

• Model size: 30 B total parameters, 3 B active per token (MoE with 8 experts).
• Data: A mixture of multilingual web crawls, high‑quality instruction datasets, and code repositories, totaling several trillion tokens.
• Compute: Trained on a cluster of NVIDIA A100 GPUs (40 GB) for several weeks, leveraging flash‑attention kernels to reduce memory bandwidth.
• Fine‑tuning: The model supports instruction‑tuning via the standard text‑generation pipeline; users can further adapt it with LoRA or full‑parameter fine‑tuning on domain‑specific corpora.
• Quantization: Post‑training quantization (PTQ) using the AWQ v1.0 method, calibrated on the NVIDIA Nemotron Post‑Training Dataset v2. This process reduces precision to 4‑bit while preserving perplexity within 0.5 % of the FP16 baseline.

The quantization step is performed with llm‑compressor, which applies group‑wise scaling (group size = 32) and per‑channel clipping to maintain numerical stability.

Related Papers

The model’s research foundation is documented in the following pre‑prints:

• GLM‑4.5 Technical Report (arXiv:2508.06471) – Provides the architectural blueprint and training methodology for the GLM‑4.x series.
• Additional references are linked via the arxiv:2508.06471 tag in the model card.

These papers describe the MoE design, flash‑attention optimizations, and the reasoning‑mode enhancements that underpin the performance of GLM‑4.7‑Flash and its quantized variant.

Licensing Information

The model card lists the license: mit under the library_name: transformers section, yet the overall “License” field is marked unknown. In practice, this means the model checkpoint itself is distributed under the MIT license, a permissive open‑source license that allows:

• Free use, modification, and distribution.
• Commercial exploitation, including embedding the model in SaaS products.
• No requirement to disclose source code of downstream projects.

Because the license is not explicitly confirmed for the quantized checkpoint, users should:

• Review the Hugging Face model card for any additional licensing notices.
• Check the files repository for a LICENSE file.
• Contact the author (Ton Cao, ton@cyan.kiwi) for clarification if the model is intended for a commercial product.

If the MIT license applies, no attribution beyond the standard “© cyankiwi” notice is required, but best practice is to credit the original authors and include a link to the model card in documentation.