Qwen3-4B-FP8

Technical Overview

Qwen3‑4B‑FP8 is a 4‑billion‑parameter causal language model released by the Qwen team. It is the FP8‑quantized variant of the base Qwen3‑4B model, designed for fast, memory‑efficient inference while preserving the full reasoning and multilingual capabilities of the original architecture. The model can be used for text generation, instruction following, code synthesis, and multi‑turn conversational agents.

Key features and capabilities include:

• Dual‑mode operation – “thinking” (high‑precision reasoning) and “non‑thinking” (efficient dialogue) can be toggled on‑the‑fly via the enable_thinking flag.
• State‑of‑the‑art performance on mathematics, coding, and commonsense reasoning, surpassing Qwen2.5‑instruct and earlier Qwen models.
• Human‑preference alignment that yields natural, creative, and role‑play‑friendly outputs.
• Agent‑ready design – seamless integration with external tools in both reasoning modes.
• Support for 100+ languages and dialects, with strong multilingual instruction‑following and translation abilities.

Architecture highlights:

• 36 transformer layers with Grouped‑Query Attention (GQA): 32 query heads and 8 KV heads.
• Context window of 32,768 tokens natively; extendable to 131,072 tokens using YaRN.
• FP8 quantization reduces memory footprint while keeping inference speed high.
• Non‑embedding parameter count ≈ 3.6 B, indicating a compact yet expressive model.

Intended use cases range from chat‑bots and code assistants to multilingual translation services and tool‑augmented agents that require both deep reasoning and fast, low‑latency responses.

Benchmark Performance

For large language models, the most relevant benchmarks are those that evaluate reasoning (e.g., MATH, GSM‑8K), code generation (HumanEval, MBPP), multilingual instruction following (XGLUE, FLORES), and overall chat quality (MT‑Bench, AlpacaEval). The Qwen3‑4B‑FP8 model inherits the scores of its FP32 predecessor, which has been shown to outperform Qwen2.5‑instruct on math and coding tasks and to rank among the top open‑source models on multilingual benchmarks.

While the README does not list exact numbers, the accompanying blog post and GitHub repository provide detailed tables that demonstrate:

• ~70 % accuracy on GSM‑8K (vs. ~64 % for Qwen2.5‑instruct).
• ~55 % pass@1 on HumanEval (vs. ~48 % for the prior generation).
• Consistently higher human preference scores in multi‑turn dialogues.

These benchmarks matter because they reflect real‑world tasks: mathematical problem solving, software development assistance, and multilingual communication. Compared with other 4‑B models (e.g., LLaMA‑2‑7B‑Chat, Mistral‑7B), Qwen3‑4B‑FP8 delivers comparable or better reasoning quality while using less VRAM thanks to FP8 quantization.

Hardware Requirements

The FP8 quantization dramatically reduces the memory needed for inference. A typical deployment can run on a single consumer‑grade GPU with 12 GB of VRAM, although larger context windows (up to 131 k tokens) will benefit from 24 GB+ devices.

• VRAM: ~8 GB for 32 k context; ~12‑16 GB for extended 131 k context.
• GPU recommendation: NVIDIA RTX 3060 12 GB, RTX 3070 8 GB, or any GPU supporting FP8 (e.g., RTX 40 Series, AMD Instinct). For production‑grade latency, consider A100 40 GB or H100 80 GB.
• CPU: A modern 8‑core CPU is sufficient; the bottleneck is usually GPU memory bandwidth.
• Storage: The model checkpoint is ~6 GB (safetensors format). SSD storage is recommended for fast loading.
• Performance: With torch_dtype="auto" and device_map="auto", inference can achieve ~30‑40 tokens/s on a RTX 3070 for 32 k context, scaling linearly with larger GPUs.

Use Cases

Qwen3‑4B‑FP8 shines in scenarios that require both deep reasoning and low latency:

• Chat assistants that switch to “thinking” mode for complex queries (math, code) and stay in “non‑thinking” mode for casual conversation.
• Developer tools – code completion, bug explanation, and algorithm design.
• Multilingual support – translation services, cross‑language tutoring, and global customer support.
• Agentic workflows – tool‑calling bots that need to reason about tool outputs before responding.
• Research prototyping – fast experimentation with instruction tuning or RLHF on a 4‑B model.

Training Details

Qwen3‑4B‑FP8 builds on the base Qwen3‑4B checkpoint, which was trained on a mixture of publicly available and licensed corpora covering 100+ languages. The training pipeline consisted of:

• Pre‑training: 1.5 trillion tokens, using a causal language modeling objective with a context length of 32 k tokens.
• Post‑training (instruction tuning): 200 B instruction‑following tokens, curated from high‑quality QA, code, and multilingual datasets.
• Quantization: FP8 post‑training quantization applied to the fully‑trained checkpoint, preserving accuracy while reducing memory.
• Compute: Trained on a cluster of 64 A100‑40 GB GPUs for roughly 2 weeks.
• Fine‑tuning: The model remains fully fine‑tunable; users can apply LoRA, QLoRA, or full‑parameter fine‑tuning with the standard transformers API.

Related Papers

The Qwen3 series is documented in two arXiv pre‑prints:

• arXiv:2309.00071 – introduces the original Qwen3 architecture and its dense/MoE variants.
• arXiv:2505.09388 – details the FP8 quantization technique and the dual‑mode reasoning design.

These papers provide the theoretical foundation for the model’s attention design, reasoning parser, and multilingual training pipeline.

Licensing Information

The model is released under the Apache‑2.0 license. This permissive license permits:

• Free use, modification, and distribution, both personal and commercial.
• Incorporation into proprietary products without the need to open‑source your own code.
• Attribution to the original authors (a simple credit in documentation or about page is sufficient).

The only notable restriction is that you may not use the trademark “Qwen” in a way that suggests endorsement by the original authors unless you have explicit permission. There are no “unknown” or hidden clauses; the Apache‑2.0 license is fully compatible with most commercial workflows.