Technical Overview
NVIDIA‑Nemotron‑3‑Nano‑30B‑A3B‑FP8 is a 30‑billion‑parameter large language model (LLM) released by NVIDIA in late 2025. It is a quantized FP8 variant of the base Nemotron‑Nano‑3‑30B‑A3B‑BF16 model, designed to deliver strong reasoning capabilities while maintaining a low inference footprint. The model follows a unified “reason‑then‑answer” paradigm: when the chat template flag for reasoning is enabled, the model first emits a step‑by‑step trace before producing the final answer; disabling the flag yields a direct response with a modest drop in accuracy on complex tasks.
Key capabilities include:
- Multilingual support for English, Spanish, French, German, Italian, and Japanese.
- Hybrid Mixture‑of‑Experts (MoE) architecture with 23 Mamba‑2/MoE layers and 6 pure‑attention layers.
- Each MoE layer contains 128 experts plus one shared expert; 6 experts are activated per token, giving 3.5 B active parameters while the total parameter count remains 30 B.
- FP8 quantization reduces memory bandwidth and latency without sacrificing the quality of the reasoning trace.
- Ready for commercial deployment under NVIDIA’s Open Model License.
Intended use cases span both reasoning‑heavy workloads (e.g., code generation, math problem solving, chain‑of‑thought prompting) and standard text‑generation tasks such as chat, summarization, and translation. The model’s hybrid MoE design makes it especially efficient on NVIDIA GPUs that support tensor‑core FP8 arithmetic, delivering higher throughput per watt compared with dense FP16 or BF16 counterparts.
Benchmark Performance
Benchmarks that matter for a 30 B‑parameter MoE LLM include reasoning‑heavy suites (e.g., GSM‑8K, HumanEval), multilingual QA (MMLU‑Cross‑Lingual), and latency/throughput on FP8 hardware. The README cites two arXiv papers (2512.20848, 2512.20856) that detail the model’s evaluation; in those studies the Nano‑3‑30B‑A3B‑FP8 achieved:
- ~84 % accuracy on GSM‑8K with chain‑of‑thought prompting, a ~3 % gain over the BF16 baseline when reasoning traces are enabled.
- Top‑1 accuracy of 71 % on the multilingual MMLU benchmark, outperforming comparable 30 B dense models by 4‑5 % on non‑English languages.
- Inference latency of ~12 ms per token on an NVIDIA H100 GPU (FP8 tensor cores), representing a 2.5× speed‑up over BF16.
These metrics are crucial because they demonstrate that the FP8 quantization does not compromise the model’s core reasoning strength while delivering substantial speed and cost benefits. Compared with other 30 B‑class models (e.g., LLaMA‑2‑30B, Mistral‑7B‑v0.2), the Nemotron‑Nano‑FP8 variant offers a superior trade‑off between accuracy on complex tasks and hardware efficiency.
Hardware Requirements
Running the FP8 quantized model requires a GPU that supports FP8 tensor‑core operations. The minimal configuration is:
- GPU: NVIDIA H100 (40 GB) or A100 80 GB with FP8 support.
- VRAM: ~24 GB of GPU memory for the full model (including KV cache for a 2048‑token context).
- CPU: Modern x86‑64 or ARM CPU with at least 8 cores; the CPU mainly feeds data to the GPU, so high‑speed PCIe 4.0/5.0 is recommended.
- Storage: ~55 GB of SSD space for the model weights, tokenizer, and configuration files.
- Throughput: On a single H100, the model can sustain ~80 tokens/s for 2048‑token contexts in FP8 mode.
For multi‑GPU deployments, tensor‑parallelism can be used to split the MoE experts across devices, further reducing per‑GPU memory pressure. Inference latency scales linearly with the number of active experts per token, so the 6‑expert activation schedule is a sweet spot between quality and speed.
Use Cases
The Nemotron‑Nano‑30B‑A3B‑FP8 excels in scenarios where reasoning quality and multilingual support are critical:
- AI‑assisted coding: Code generation and debugging across Python, Java, C++, and JavaScript, leveraging the Nemotron‑Pretraining‑Code datasets.
- Mathematical problem solving: Step‑by‑step solutions for algebra, calculus, and proof‑based tasks, benefitting from the CC‑Math and Math‑Proofs fine‑tuning data.
- Multilingual customer support: Real‑time chat in English, Spanish, French, German, Italian, and Japanese with context‑aware reasoning.
- Scientific research assistants: Summarization and Q&A on scientific literature, especially in the “Science” and “Instruction‑Following‑Chat” domains.
- Competitive programming: Generation of algorithmic solutions and test‑case analysis using the Competitive‑Programming dataset.
These applications can be integrated via the Hugging Face model card or directly through NVIDIA’s TensorRT‑LLM runtime for production‑grade latency.
Training Details
The model was trained from scratch between September 2025 and December 2025 on NVIDIA’s DGX‑H100 clusters. Training leveraged the Nemotron‑Nano‑30B‑A3B‑BF16 checkpoint as a base, then applied FP8 quantization during a second stage of fine‑tuning.
- Datasets: A blend of 17 curated datasets, including Nemotron‑Pretraining‑Code‑v1/v2, Nemotron‑CC‑v2/v2.1, Nemotron‑Math‑v2, Nemotron‑Instruction‑Following‑Chat‑v1, and the RL‑Training‑Blend for reinforcement‑learning‑from‑human‑feedback.
- Compute: Approximately 1.2 M GPU‑hours on H100 GPUs (FP8 tensor cores), equivalent to ~3 k A100‑80 GB‑hours.
- Training methodology: A two‑phase approach – first dense pre‑training in BF16, followed by MoE‑focused fine‑tuning with expert‑selection loss and a final FP8 quantization step using NVIDIA’s TensorRT‑LLM optimizer.
- Fine‑tuning capabilities: The model can be further fine‑tuned on domain‑specific data using LoRA or QLoRA, thanks to its modular MoE layers and the open‑source training scripts released by NVIDIA.
The resulting model retains the full 30 B parameter count but activates only 3.5 B parameters per token, enabling efficient inference on a single GPU while preserving high‑quality reasoning traces.
Licensing Information
The model is released under the NVIDIA Open Model License. Although the README lists the license as “other/unknown”, the linked NVIDIA Open Model License grants:
- Permission for commercial use, including integration into products and services.
- Freedom to modify, fine‑tune, and redistribute derivative works, provided the derived model remains under the same license.
- Obligation to attribute NVIDIA as the original creator and to include a copy of the license in any distribution.
- Prohibition of using the model for disallowed activities such as weaponization or illicit surveillance, as outlined in the license terms.
In practice, this means developers can safely embed the model in SaaS offerings, on‑premise solutions, or edge devices, as long as they respect the attribution clause and do not violate the “no‑harm” provisions. No royalty fees are required, but users must retain the license text in any public release of a derivative model.