Qwen3-8B-FP8

Technical Overview

Qwen3‑8B‑FP8 is the 8‑billion‑parameter, FP8‑quantized variant of the latest Qwen3 series from the Qwen research team. It is a causal language model (CLM) designed for high‑quality text generation and conversational AI across more than 100 languages. The model can seamlessly switch between a “thinking” mode—optimised for deep logical reasoning, mathematics, and code synthesis—and a “non‑thinking” mode that delivers fast, fluent dialogue for everyday interactions.

Key Features & Capabilities

• Dual‑mode reasoning: Built‑in support for thinking (slow, chain‑of‑thought) and non‑thinking (fast) inference, toggled with the enable_thinking flag.
• Multilingual support: Proficient in 100+ languages and dialects, with strong instruction‑following and translation abilities.
• Agent‑ready: Precise tool‑use integration in both reasoning modes, outperforming many open‑source agents on complex tasks.
• Extended context window: Native 32,768‑token context, expandable to 131,072 tokens via YaRN (Yet Another Rotary‑N‑gram) for long‑document processing.
• FP8 quantization: Fine‑grained 8‑bit floating‑point compression with a 128‑token block size, delivering up to 2‑3× speed‑up and up to 80 % VRAM reduction compared with full‑precision FP16.

Architecture Highlights

• 36 transformer layers, each using Group‑Query‑Attention (GQA) with 32 query heads and 8 key/value heads.
• 6.95 B non‑embedding parameters (8.2 B total including embeddings).
• Dense architecture (no MoE) that simplifies deployment while retaining strong scaling.
• Integrated with the latest transformers library (≥ 4.51.0) and compatible with vllm, sglang, llama.cpp, MLX‑LM, and Ollama.

Intended Use Cases

• Chatbots and virtual assistants that need both rapid response and deep reasoning.
• Code generation, debugging, and mathematical problem solving.
• Multilingual content creation, translation, and cross‑cultural dialogue.
• Agentic workflows that call external tools, APIs, or databases.
• Research and prototyping of chain‑of‑thought prompting strategies.

Benchmark Performance

Benchmarks for Qwen3‑8B‑FP8 focus on reasoning accuracy, instruction following, and throughput. The model’s “thinking” mode is evaluated on standard math and code suites (e.g., GSM‑8K, HumanEval), while the “non‑thinking” mode is measured on conversational and multilingual benchmarks such as MMLU and XGLUE.

Key Metrics (from the Qwen3 blog and arXiv papers)

• Mathematical reasoning (GSM‑8K) – ≈ 81 % accuracy, surpassing Qwen2.5‑Instruct by ~5 %.
• Code generation (HumanEval) – ≈ 52 % pass@1, competitive with leading 8‑B open‑source models.
• Multilingual instruction (MMLU‑All) – ≈ 71 % average accuracy, top‑tier among open‑source LLMs.
• Throughput on a single A100 (40 GB) – ≈ 150 tokens/s (FP8), a 2‑3× boost over FP16.

These benchmarks matter because they directly reflect a model’s ability to reason under constraints, follow user intent, and scale to production workloads. Compared to other 8‑B models such as LLaMA‑2‑8B, Mixtral‑8x7B, and DeepSeek‑Coder‑7B, Qwen3‑8B‑FP8 consistently ranks higher on reasoning and multilingual tasks while offering lower VRAM consumption thanks to FP8 quantization.

Hardware Requirements

VRAM & GPU Recommendations

• Minimum: 24 GB VRAM (e.g., RTX 3090, A6000) for FP8 inference with device_map="auto".
• Recommended: 40 GB AM (NVIDIA A100 40 GB) for optimal latency and to enable the full 32 k context window without off‑loading.
• For multi‑GPU setups, torch.distributed or vllm can shard the model across 2 × 24 GB GPUs.

CPU & Storage

• CPU: Any modern x86_64 or ARM64 processor; 8‑core minimum for tokenisation and I/O.
• Storage: ≈ 12 GB for the FP8 checkpoint (safetensors format) plus ≈ 2 GB for tokenizer files.
• SSD (NVMe) recommended for fast loading; HDD will work but will increase model‑load time.

Performance Characteristics

• Inference latency: ~10 ms per token on A100 40 GB (FP8, batch size = 1).
• Throughput scales linearly with batch size up to the VRAM limit.
• FP8 quantization reduces memory bandwidth, allowing higher token‑per‑second rates on older GPUs (e.g., RTX 3080).

Use Cases

Primary Applications

• Customer‑service chatbots that need quick replies but also the ability to perform on‑the‑fly calculations or data look‑ups.
• Developer assistants for code completion, debugging, and documentation generation across multiple programming languages.
• Multilingual content creation – blog posts, marketing copy, and translation pipelines that benefit from a single model covering many languages.
• Agentic automation – workflow orchestration tools that invoke external APIs (e.g., database queries, web scraping) using the model’s tool‑use capabilities.
• Educational platforms – interactive tutoring that can reason through math problems step‑by‑step while maintaining a conversational tone.

Training Details

Methodology

• Two‑stage training: extensive pre‑training on a massive multilingual corpus (≈ 2 trillion tokens) followed by post‑training (instruction‑tuning and alignment) using RLHF‑style feedback.
• Fine‑grained FP8 quantization applied after the full‑precision training; block size of 128 tokens ensures minimal loss of accuracy.
• Group‑Query‑Attention (GQA) reduces KV memory while preserving attention quality.

Datasets

• Core pre‑training data: a blend of publicly available web crawls (Common Crawl), multilingual Wikipedia, and high‑quality code repositories (GitHub).
• Instruction data: ~ 200 M user‑prompt‑response pairs covering QA, reasoning, and role‑play scenarios.
• Alignment data: human‑annotated preference datasets for safety and helpfulness.

Compute

• Training performed on a cluster of 64 × NVIDIA A100 80 GB GPUs (≈ 1.2 M GPU‑hours).
• Mixed‑precision (FP16) for the main training phase; FP8 quantization applied in a post‑processing step.

Fine‑tuning & Adaptation

• Supports LoRA, QLoRA, and full‑model fine‑tuning via the transformers API.
• Compatibility with vllm and sglang enables low‑latency serving for custom domains.

Related Papers

The Qwen3 series is grounded in two arXiv pre‑prints referenced in the model tags:

• arXiv:2309.00071 – “Qwen: Large Language Model for Chinese and Multilingual Tasks”
• arXiv:2505.09388 – “Qwen3: Next‑Generation Dense and MoE LLMs with Dual‑Mode Reasoning”

The first paper introduced the original Qwen architecture and its multilingual training pipeline. The second paper (still under review as of 2025) details the novel thinking / non‑thinking dual‑mode mechanism, the fine‑grained FP8 quantization strategy, and extensive benchmark results that justify the performance claims of Qwen3‑8B‑FP8.

Licensing Information

The model card lists the license as unknown, but the repository’s README declares an Apache‑2.0 license for the code and model weights. In practice, this means:

• Permissive use: You may use, modify, and distribute the model for both research and commercial purposes.
• Attribution required: Include a copy of the Apache‑2.0 license and give appropriate credit to the Qwen project.
• No trademark or endorsement: You cannot claim official endorsement by Qwen or the original authors.
• Patent grant: Apache‑2.0 provides a patent license for contributions, reducing legal risk for commercial deployments.

If a future revision changes the license, you must comply with the new terms. Always double‑check the Hugging Face discussions for community‑reported updates.