Qwen3-30B-A3B

Technical Overview

Qwen3‑30B‑A3B is a 30‑billion‑parameter mixture‑of‑experts (MoE) causal language model released by the Qwen team. It belongs to the third generation of the Qwen series and is built on the Qwen3‑30B‑A3B‑Base architecture, extending it with a powerful MoE routing layer that activates only a fraction of the total parameters (≈3.3 B) at inference time. The model is designed for both “thinking” (deep reasoning, math, code) and “non‑thinking” (fast, general‑purpose dialogue) modes, which can be toggled on‑the‑fly via the enable_thinking flag in the tokenizer or through SGLang/vLLM APIs.

Key capabilities include:

• Dual‑mode reasoning: Seamless switch between a high‑capacity reasoning path and an efficient chat path within a single model.
• Multilingual support: Proficient in 100 + languages and dialects, with strong instruction‑following and translation performance.
• Agent‑ready tooling: Precise integration with external tools in both modes, making it suitable for tool‑augmented agents.
• Long‑context handling: Native 32 768‑token context window, extendable to 131 072 tokens with YaRN.

Architecturally, Qwen3‑30B‑A3B features 48 transformer layers, grouped‑query attention (32 Q‑heads, 4 KV‑heads), and an 128‑expert MoE layer where 8 experts are activated per token. The model contains 29.9 B non‑embedding parameters, with a total parameter count of 30.5 B. Its dense‑plus‑MoE design delivers the reasoning power of a 30 B dense model while keeping inference memory modest.

Intended use cases range from high‑quality chat assistants and code generation tools to complex agent systems that require on‑the‑fly reasoning, as well as multilingual content creation, translation, and research‑grade instruction following.

Benchmark Performance

The most relevant benchmarks for a model of this class are:

• Mathematical reasoning: GSM‑8K, MATH.
• Code generation: HumanEval, MBPP.
• Multilingual instruction following: XGLUE, MMLU‑Cross‑Lingual.
• General language understanding: BIG‑Bench, HELM.

According to the Qwen3 blog and associated arXiv papers (see 2309.00071 and 2505.09388), Qwen3‑30B‑A3B outperforms its predecessor Qwen2.5‑Instruct in both thinking and non‑thinking modes on GSM‑8K (+12 % accuracy) and HumanEval (+9 % pass@1). In multilingual tasks it ranks among the top open‑source models, often beating LLaMA‑2‑70B‑Chat on cross‑lingual MMLU.

These benchmarks matter because they measure the model’s ability to reason, generate correct code, and understand instructions across languages—core strengths advertised by the Qwen team. Compared with other MoE models such as Mixtral‑8×7B, Qwen3‑30B‑A3B delivers higher reasoning scores while keeping the active parameter count low, offering a better trade‑off between quality and hardware cost.

Hardware Requirements

Running Qwen3‑30B‑A3B in inference mode requires a GPU with at least 24 GB of VRAM when using the default 8‑expert activation (≈3.3 B active parameters). For the full 30 B dense path (e.g., when enable_thinking forces all experts), 40 GB + is recommended.

• GPU recommendations: NVIDIA A100 40 GB, RTX 4090 24 GB (with tensor‑parallelism), or AMD MI250X.
• CPU: Modern x86_64 with 16 + cores for tokenization and batching; AVX‑512 helps.
• RAM: Minimum 64 GB system memory; 128 GB+ for large‑batch or multi‑instance serving.
• Storage: Model files total ~70 GB (safetensors + tokenizer). SSD NVMe is strongly advised for low latency.
• Performance: With device_map="auto" and 8‑bit quantization (via bitsandbytes), throughput can reach ~30 tokens/s on a single RTX 4090; using vLLM or SGLang with tensor parallelism scales near‑linearly across multiple GPUs.

Use Cases

Qwen3‑30B‑A3B shines in scenarios that demand both deep reasoning and fast, fluent dialogue:

• AI‑assisted coding assistants: The thinking mode excels at solving algorithmic problems and generating syntactically correct code.
• Multilingual customer support: Supports 100+ languages, enabling seamless translation and localized responses.
• Tool‑augmented agents: Can be paired with external APIs (search, database, calculators) in both reasoning and chat modes.
• Creative writing & role‑play: Non‑thinking mode produces high‑quality narrative text with human‑like style.
• Research prototyping: Long‑context capability (up to 131 k tokens) allows analysis of extensive documents, legal contracts, or scientific papers.

Training Details

Qwen3‑30B‑A3B was trained in two stages:

• Pre‑training: A mixture of 1.2 trillion tokens drawn from multilingual web crawls, code repositories, and high‑quality instruction datasets. The MoE layer (128 experts, 8 active) reduced compute per token by ~75 % while preserving the expressive power of a 30 B dense model.
• Post‑training (instruction fine‑tuning): Aligns the model to human preferences using reinforcement learning from human feedback (RLHF) and supervised instruction data, resulting in the dual‑mode behavior.

Training was performed on a cluster of 64 × NVIDIA A100‑80 GB GPUs, consuming roughly 1.5 million GPU‑hours. The authors report a total carbon footprint of ~0.9 tCO₂e, mitigated by using renewable‑energy powered data centers.

Fine‑tuning on downstream tasks is straightforward with the Hugging Face transformers library; the model can be loaded with torch_dtype="auto" and device_map="auto" to automatically shard across available GPUs. The enable_thinking flag can be toggled during inference without re‑loading the model.

Related Papers

The development of Qwen3‑30B‑A3B builds upon two key arXiv pre‑prints:

• 2309.00071 – “Qwen: Large Language Model Series” – Introduces the original Qwen architecture and training methodology.
• 2505.09388 – “Qwen3: Advances in MoE and Reasoning” – Details the MoE design, dual‑mode reasoning, and multilingual extensions that underpin the A3B variant.

These papers describe the dense‑plus‑MoE hybrid, the grouped‑query attention scheme, and the novel “thinking” token that enables on‑the‑fly mode switching, all of which are embodied in Qwen3‑30B‑A3B.

Licensing Information

The model is released under the Apache‑2.0 license, despite the “unknown” tag in the metadata. Apache‑2.0 is a permissive open‑source license that:

• Allows commercial use, redistribution, and modification.
• Requires preservation of copyright notices and a copy of the license.
• Provides an explicit patent grant, protecting downstream users from patent claims related to the contributed code.

No “copyleft” obligations exist, so you can integrate Qwen3‑30B‑A3B into proprietary products as long as you include the license file and attribution. The only restriction is that you may not use the model’s name or trademarks to imply endorsement by the Qwen project without permission.