Technical Overview
GadflyII/GLM‑4.7‑Flash‑NVFP4 is a 30‑billion‑parameter, Mixture‑of‑Experts (MoE) language model derived from the zai‑org/GLM‑4.7‑Flash base. It employs a novel NVFP4 mixed‑precision quantization that stores the majority of its weights in 4‑bit floating‑point (FP4, E2M1) while keeping attention, norm, gate and embedding layers in BF16. This hybrid scheme preserves most of the original BF16 accuracy (only a ≈ 1.3 % drop on MMLU‑Pro) while shrinking the model size from 62.4 GB to about 20.4 GB – a 3.1× compression factor that fits comfortably on a single high‑end GPU.
Key capabilities include:
- Multilingual generation in English and Chinese.
- 30 B total parameters with a 3 B active‑expert footprint per token (64 experts, 4 active + 1 shared).
- Long context length up to 202 752 tokens, enabling extended conversations or document‑level reasoning.
- Fast inference via vLLM’s NVFP4 support, which can exploit FP4 tensor cores on NVIDIA Blackwell, Hopper and Ada Lovelace GPUs.
Architecturally the model follows the Glm4MoeLiteForCausalLM design: a stack of 47 transformer layers where the feed‑forward network (MLP) is split into dense and expert sub‑modules. The dense MLP and each expert’s MLP are quantized to FP4, while the multi‑head low‑rank attention (MLA) stays in BF16 to protect the sensitive Q/K/V projections. Norms, gates and embeddings also remain in BF16, following best‑practice mixed‑precision guidelines.
Intended use cases revolve around high‑throughput, cost‑effective text generation: chat assistants, code completion, summarisation, and multilingual content creation where a 30 B‑scale MoE model is desirable but memory constraints would otherwise be prohibitive.
Benchmark Performance
The most relevant benchmark for a MoE language model of this size is MMLU‑Pro, a comprehensive multi‑task language understanding suite. The NVFP4‑quantized version of GLM‑4.7‑Flash achieves 23.55 % overall accuracy, only 1.28 % lower than the full‑precision BF16 baseline (24.83 %). Compared to a uniform FP4 quantization (16.84 % accuracy), the mixed‑precision approach delivers a +6.71 % absolute gain while keeping the model size under 21 GB.
Category‑level results show a consistent, modest drop across Social Sciences, Humanities, STEM and other domains, with the most pronounced loss in Computer Science (‑4.64 %). Notably, the model even outperforms BF16 on Business (+0.63 %) and Math (+0.23 %) categories, illustrating the effectiveness of the selective BF16 preservation for sensitive components.
These numbers matter because they demonstrate that the NVFP4 quantization retains most of the original model’s reasoning power while dramatically reducing memory and bandwidth requirements, a trade‑off that is critical for production deployments on a single GPU.
Hardware Requirements
GPU memory: The model’s compressed size is ~20.4 GB. For inference you should allocate at least 24 GB of VRAM to accommodate activation buffers, KV‑cache for the 202 752‑token context, and the vLLM runtime overhead. A single NVIDIA RTX 4090 (24 GB) or an RTX 6000 Ada (48 GB) is sufficient; multi‑GPU setups can be used for larger batch sizes or higher throughput.
GPU architecture: NVFP4 quantization requires tensor cores that support FP4 (E2M1) arithmetic – currently available on Blackwell (RTX 50/Pro), Hopper (RTX 6000 Ada), and Ada Lovelace GPUs. Older architectures will fall back to BF16 or will not be able to load the model.
CPU: A modern 8‑core CPU (e.g., AMD Ryzen 7 5800X or Intel i7‑12700K) is adequate for token‑level preprocessing and managing the vLLM server. The CPU does not need to match the GPU’s compute power.
Storage: The model files (including compressed‑tensors, tokenizer, and config) occupy roughly 20 GB on disk. SSD storage (NVMe preferred) is recommended to keep loading times low; a 100 GB free space margin is safe for additional checkpoints or fine‑tuning data.
Performance: On a single RTX 4090 with vLLM 0.14+, the model can generate ~30 tokens/second for a 4 k context at temperature 0.7, scaling linearly with additional GPUs when tensor‑parallelism is enabled.
Use Cases
The GLM‑4.7‑Flash‑NVFP4 model shines in scenarios that demand high‑quality multilingual generation while staying within a single‑GPU memory budget.
- Chatbots & virtual assistants: Bilingual (EN/ZH) conversational agents that can handle long dialogues thanks to the 202 k token context.
- Document summarisation: Summarise lengthy reports, legal contracts, or academic papers without truncation.
- Code assistance: Generate code snippets or explain programming concepts in both English and Chinese.
- Content localisation: Translate marketing copy or product descriptions while preserving style.
- Educational tutoring: Answer subject‑specific questions (e.g., biology, economics) with the model’s demonstrated competence on MMLU‑Pro.
Integration is straightforward via the vllm server or the transformers library, allowing deployment as a REST endpoint, a local CLI tool, or a cloud‑native microservice.
Training Details
The model inherits the training pipeline of zai‑org/GLM‑4.7‑Flash, a 30 B‑parameter MoE model trained on a massive multilingual corpus (English + Chinese) using a mixture of supervised and self‑supervised objectives. The original training employed a dense‑to‑MoE conversion, 64 experts, and a 4‑active‑expert routing per token, yielding a 202 k token context window.
For the NVFP4 version, a custom quantization and calibration script (based on neuralmagic/calibration) was applied post‑training. Calibration used 128 random samples with a maximum sequence length of 2048 tokens, covering all 64 experts to compute per‑expert scaling factors. The quantization kept MLP experts and the first dense MLP in FP4, while attention, norms, gates and embeddings remain in BF16. No additional fine‑tuning was performed after quantization.
Fine‑tuning is possible via the glm4_moe_lite architecture in transformers 5.0+. Users can load the model with trust_remote_code=True and continue training on domain‑specific data, benefiting from the same mixed‑precision representation.
Licensing Information
The model inherits the Apache 2.0 license from its base model zai‑org/GLM‑4.7‑Flash. Although the README lists the license as “unknown”, the explicit “license: apache-2.0” tag confirms that the same permissive terms apply.
Under Apache 2.0 you may:
- Use the model for commercial or non‑commercial purposes.
- Modify, redistribute, or embed the model in proprietary software.
- Publish derived works without requiring the source code to be open.
The only obligations are:
- Provide proper attribution to the original authors (Zhipu AI) and to GadflyII for the NVFP4 quantization.
- Include a copy of the Apache 2.0 license in any distribution.
- State any modifications made to the original model.
There are no patent or trademark restrictions that would block commercial deployment, making the model suitable for SaaS products, on‑premise solutions, and research projects alike.