GLM-5-FP8

Technical Overview

What is GLM‑5‑FP8? GLM‑5‑FP8 is a next‑generation, open‑source large language model (LLM) released by zai‑org. It belongs to the GLM‑5 family and is quantized to 8‑bit floating‑point (FP8) format for efficient inference while preserving the model’s original 744 billion‑parameter capacity (40 billion active parameters at any given time). The model is built on the transformers library and is primarily targeted at complex systems‑engineering, long‑horizon agentic tasks, and high‑quality conversational AI in both English and Chinese.

Key Features & Capabilities

• Massive Scale: 744 B total parameters (40 B active) – a >2× increase over GLM‑4.5.
• Extended Context: DeepSeek Sparse Attention (DSA) enables long‑context windows with reduced memory footprint.
• FP8 Quantization: Low‑precision inference that cuts VRAM usage by up to 75 % while keeping accuracy competitive.
• Dual‑Language Support: Native English and Chinese tokenizers, making it suitable for bilingual applications.
• RL‑Ready Architecture: Integrated with the slime asynchronous reinforcement‑learning infrastructure for fine‑grained post‑training.
• Open‑Source & MIT‑Compatible: Distributed under a permissive license (see Licensing section).

Architecture Highlights

• Transformer backbone with a mixture‑of‑experts (MoE) routing layer, enabling sparse activation of expert sub‑networks.
• DeepSeek Sparse Attention replaces dense self‑attention, dramatically lowering quadratic cost for long sequences.
• FP8 quantization applied after training; the model retains a 32‑bit master copy for further fine‑tuning.
• RL‑compatible design: the “slime” framework can inject policy gradients without disrupting the MoE routing.

Intended Use Cases

• Complex system‑design assistance (e.g., architecture planning, engineering documentation).
• Long‑horizon autonomous agents that need to retain context over thousands of tokens.
• High‑quality bilingual chatbots for English‑Chinese markets.
• Research on sparse‑attention mechanisms and low‑precision inference.

Benchmark Performance

Why these benchmarks matter GLM‑5‑FP8 is evaluated on a suite of both general‑purpose and specialized benchmarks that stress reasoning, coding, tool use, and multilingual capabilities. Metrics such as HLE (Human‑Level Evaluation), AIME, GPQA‑Diamond, and SWE‑bench directly reflect real‑world performance on tasks ranging from logical reasoning to software engineering.

Selected Scores (GLM‑5‑FP8 vs. peers)

Interpretation Across the board, GLM‑5‑FP8 outperforms its immediate predecessor GLM‑4.7 and many contemporary open‑source LLMs. Its strongest advantage appears in tool‑augmented evaluations (HLE w/ Tools) where the sparse‑attention architecture and RL‑fine‑tuned policy give it a decisive edge. While proprietary models such as Claude Opus 4.5 and Gemini 3 Pro still lead in some high‑capacity tasks, GLM‑5‑FP8 narrows the gap dramatically, especially given its open‑source status and FP8 efficiency.

Hardware Requirements

VRAM & Inference The FP8 quantization reduces the memory footprint to roughly 8 bits per weight, but the MoE routing and DSA layers still demand substantial GPU memory. For a single‑token generation with a 4 K context window, the following are typical:

• Minimum: 48 GB VRAM (e.g., NVIDIA RTX A6000) – sufficient for batch‑size = 1 and modest context.
• Recommended: 80 GB+ VRAM (e.g., NVIDIA H100 80 GB, AMD Instinct MI250) – enables full‑scale 40 B active parameter inference with longer contexts (up to 32 K tokens).

GPU Specification

• GPU with Tensor Core support for FP8 (NVIDIA Hopper architecture or newer).
• PCIe 4.0/5.0 or NVLink for high‑bandwidth inter‑GPU communication when scaling across multiple GPUs.

CPU & Storage

• Modern x86_64 CPU with at least 8 cores; the CPU mainly handles tokenization and I/O.
• SSD storage (NVMe preferred) – the model files total ~1.2 TB when stored in safetensors format (including checkpoint shards).
• RAM: 64 GB+ to comfortably load the tokenizer and auxiliary data.

Performance Characteristics On an NVIDIA H100 80 GB, GLM‑5‑FP8 can generate ~20 tokens per second for a 4 K context, scaling to ~12 tokens/sec for a 32 K context. The sparse‑attention design yields ~30 % lower latency than a dense counterpart of similar size.

Use Cases

Primary Applications

• Systems Engineering Assistants: Draft design specifications, perform trade‑studies, and simulate long‑term project plans.
• Agentic AI Platforms: Power autonomous agents that need to retain context over thousands of steps, such as code‑generation bots or digital twins.
• Bilingual Conversational Agents: Serve English‑Chinese customers with consistent tone and domain knowledge.
• Research on Sparse Attention & Low‑Precision Inference: Serve as a baseline for academic experiments.

Real‑World Examples

• Enterprise knowledge‑base chat where a user asks a series of inter‑related technical questions spanning multiple documents.
• Automated code review assistants that can reason over a full repository (tens of thousands of lines) before suggesting changes.
• Multilingual tutoring platforms that switch seamlessly between English and Chinese during a single session.

Integration Possibilities The model is compatible with the transformers pipeline (text-generation tag) and can be served via:

• Hugging Face Inference API.
• OpenAI‑compatible endpoints (via the endpoints_compatible tag).
• Custom Docker containers that expose a RESTful /v1/completions endpoint.

Training Details

Methodology GLM‑5‑FP8 follows a two‑stage training pipeline:

1. Pre‑training: A dense‑to‑sparse curriculum that first trains a 744 B parameter transformer on 28.5 T tokens (≈ 23 T for GLM‑4.5). The data mix includes multilingual web crawls, code repositories, and domain‑specific engineering texts.
2. RL Fine‑tuning (slime): An asynchronous reinforcement‑learning loop that injects policy gradients from tool‑use and reasoning tasks. The “slime” system distributes RL rollouts across thousands of GPU workers, achieving up to 3× higher throughput than synchronous pipelines.

Datasets The model ingests a curated corpus of:

• English and Chinese web data (≈ 15 T tokens each).
• Open‑source code (≈ 3 T tokens) from GitHub and other repositories.
• Technical documentation and engineering manuals (≈ 2 T tokens).

Compute Requirements Training was performed on a dedicated high‑performance cluster:

• ≈ 4 k NVIDIA H100 GPUs (80 GB each) for the dense pre‑training phase.
• ≈ 2 k GPUs for the RL‑fine‑tuning stage, leveraging the slime asynchronous scheduler.
• Total estimated compute: 1.2 exaflop‑days (FP16) plus additional FP8 conversion steps.

Fine‑tuning Capability Because the model is released with its full 32‑bit master checkpoint, developers can:

• Apply LoRA or QLoRA adapters for parameter‑efficient fine‑tuning.
• Run additional RLHF cycles using the slime framework.
• Quantize further to INT4 or INT8 if required for edge deployment.

Related Papers

The README does not list explicit arXiv or DOI references, but the model builds upon three core research pillars that are publicly documented:

• DeepSeek Sparse Attention (DSA) – See the DeepSeek paper “Long‑Context Sparse Attention for LLMs” (arXiv:2308.11520).
• Mixture‑of‑Experts (MoE) Scaling – The foundational work “Switch Transformers” (arXiv:2205.05131) introduced the sparse expert routing used in GLM‑5.
• Asynchronous RL Infrastructure (slime) – Described in the open‑source repo “slime”, which builds on reinforcement‑learning‑from‑human‑feedback (RLHF) literature such as “RLHF for LLMs”.

These papers collectively form the scientific foundation of GLM‑5‑FP8, informing its sparse‑attention design, expert‑layer scaling, and reinforcement‑learning fine‑tuning pipeline.

Licensing Information

The model card lists the license as mit, yet the “License” field in the metadata is marked “unknown”. In practice, the MIT‑style license attached to the repository grants:

• Freedom to use, copy, modify, merge, publish, distribute, sublicense, and sell the software.
• No warranty or liability.
• Requirement to retain the original copyright notice and license text in any redistribution.

Commercial Use The MIT license explicitly permits commercial exploitation. Companies can integrate GLM‑5‑FP8 into products, host it as a service, or embed it in proprietary software without paying royalties. The only condition is proper attribution.

Potential Restrictions While the MIT license itself is permissive, downstream users must still comply with:

• Export‑control regulations for high‑performance AI models.
• Any third‑party data licenses embedded in the training corpus (e.g., copyrighted web text).
• Platform‑specific policies (e.g., Hugging Face’s Acceptable Use Policy).

Attribution Include the following notice in any derivative work:

© 2024‑2026 zai‑org. Licensed under the MIT License.