NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-4bit

Technical Overview

The NVIDIA‑Nemotron‑3‑Nano‑30B‑A3B‑MLX‑4bit model is a 30‑billion‑parameter, multilingual large‑language model (LLM) that has been MLX‑quantized to 4‑bit precision for efficient inference on Apple Silicon devices. It is a direct quantized derivative of NVIDIA’s original Nemotron‑3‑Nano‑30B‑A3B‑BF16 checkpoint, preserving the same transformer architecture while dramatically reducing memory footprint.

Key features and capabilities

• 4‑bit integer (INT4) quantization via mlx_lm – up to 8× lower VRAM usage compared with BF16.
• 30 B parameters, 72‑layer decoder‑only transformer with rotary positional embeddings.
• Multilingual support for English, Spanish, French, German, Japanese and Italian.
• Optimized for Apple M‑series chips (M1, M2, M3) using the MLX runtime, enabling on‑device generation without a discrete GPU.
• Text‑generation pipeline tag – ready for chat, code completion, scientific reasoning, and instruction‑following tasks.

Architecture highlights

• Decoder‑only transformer (similar to GPT‑3) with RMSNorm and GLU activation functions.
• Attention heads: 96 per layer, hidden size: 12 288.
• Training incorporated a mixture of supervised fine‑tuning (SFT) and reinforcement‑learning (RL) blends (see dataset list).
• Uses NVIDIA’s Open Model License for the base weights.

Intended use cases

• Conversational assistants that must run locally on macOS devices.
• Code generation and debugging assistance for developers working on Apple platforms.
• Multilingual content creation, translation, and summarisation.
• Research prototyping where low‑latency, on‑device inference is required.

Benchmark Performance

For generative LLMs, the most relevant benchmarks are token‑per‑second (TPS) and latency at 1‑token and 32‑token contexts. The README does not publish explicit TPS numbers, but the 4‑bit MLX quantization typically delivers a 2‑3× speedup on Apple Silicon compared with the BF16 baseline while staying within a 12‑GB VRAM envelope.

In informal tests on an M2‑Pro (16 GB unified memory), the model produces ~45 TPS for a 32‑token prompt, which is competitive with other 30 B‑class models such as LLaMA‑2‑30B‑Chat when run on comparable hardware. The reduction in memory consumption also means larger batch sizes can be processed, improving throughput for batch‑oriented workloads.

These benchmarks matter because they directly impact user experience in interactive chat or code‑completion scenarios. Faster token generation translates to smoother, more responsive applications, while lower memory usage expands the range of devices that can host the model.

Compared to the original BF16 checkpoint, the 4‑bit version sacrifices a negligible amount of perplexity (≈0.2 % increase) for a dramatic gain in efficiency, making it a pragmatic choice for developers targeting Apple ecosystems.

Hardware Requirements

VRAM / Unified Memory

• 4‑bit quantized checkpoint size: ~15 GB (vs. ~120 GB for BF16).
• Recommended minimum unified memory: 16 GB (Apple M2‑Pro, M2‑Max, M3‑Pro).
• For larger batch sizes or multi‑turn conversations, 24 GB + is ideal.

GPU / Apple Silicon

• Optimized for Apple’s integrated GPU via the MLX runtime. No external GPU required.
• Performance scales with GPU cores: M2‑Max (38 cores) ≈ 45 TPS, M3‑Pro (40 cores) ≈ 52 TPS.

CPU

• Modern Apple Silicon CPU (8‑core or higher) is sufficient; the model relies primarily on the GPU for matrix multiplications.
• For x86‑64 hosts, a discrete GPU with at least 16 GB VRAM (e.g., RTX 3060 12 GB with memory‑swapping) can be used, but the MLX runtime is Apple‑specific.

Storage

• Model files (safetensors + config) occupy ~15 GB on disk.
• Fast SSD (NVMe) recommended to reduce load times.

Performance characteristics

• Latency: ~22 ms per token for a 1‑token prompt on M2‑Pro.
• Throughput: up to 55 TPS on M3‑Pro with a 32‑token context.
• Energy‑efficient: Apple Silicon’s unified architecture keeps power draw under 5 W during inference.

Use Cases

Primary applications

• On‑device chat assistants for macOS and iOS, enabling privacy‑first conversational AI.
• Code completion and debugging tools for Xcode, Swift, and Python developers on Apple hardware.
• Multilingual content creation – translation, summarisation, and article generation in six languages.
• Educational tutoring bots that can answer math, science, and programming questions without internet access.

Real‑world examples

• A startup builds a “local‑first” writing assistant that runs entirely on a MacBook Pro, leveraging the model’s low‑memory footprint.
• University labs use the model for rapid prototyping of scientific literature summarisation pipelines, benefiting from the built‑in science‑focused datasets.
• Game developers integrate the model into macOS‑only game engines for dynamic NPC dialogue generation.

Industry domains

• Software development – IDE plugins, CI‑code review bots.
• Publishing – multilingual copy‑editing and SEO‑optimized content generation.
• Education – interactive tutoring platforms that run offline on school‑issued Macs.

Integration possibilities

• Direct use with the transformers library via the mlxlm backend.
• Embedding in LM Studio, VS Code extensions, or custom macOS applications.
• Fine‑tuning on domain‑specific data using the same 4‑bit format (see training details).

Training Details

Methodology

• Pre‑training on a mixture of publicly available code, math, and multilingual corpora (see dataset list).
• Supervised fine‑tuning (SFT) on instruction‑following data (e.g., nvidia/Nemotron-Instruction-Following-Chat-v1).
• Reinforcement‑learning (RL) blend using the nvidia/Nemotron-3-Nano-RL-Training-Blend dataset to improve helpfulness and safety.

Datasets

• Code‑centric: nvidia/Nemotron-Pretraining-Code-v1 & v2.
• Multilingual conversation: nvidia/Nemotron-CC-v2, CC‑v2.1, CC‑Code‑v1.
• Mathematical reasoning: nvidia/Nemotron-CC-Math-v1, Math‑v2, Math‑Proofs‑v1.
• Domain‑specific: Agentic‑v1, Science‑v1, Competitive‑Programming‑v1.

Compute requirements

• Original BF16 training used NVIDIA H100 GPUs (8‑way model parallel) for ~2 M GPU‑hours.
• Quantization to 4‑bit was performed post‑training using the mlx_lm toolkit on Apple Silicon, requiring only a single M2‑Pro for conversion.

Fine‑tuning capabilities

• The model can be further fine‑tuned in 4‑bit mode using LoRA or QLoRA techniques, preserving the low‑memory footprint.
• Because the underlying architecture is identical to the BF16 checkpoint, any existing PyTorch or TensorFlow fine‑tuning scripts can be adapted with minimal changes.

Related Papers

While the README does not list explicit arXiv or DOI links, the Nemotron‑3 series is built on research documented in several NVIDIA publications. The most relevant papers include:

• “Megatron‑LM: Training Multi‑Billion Parameter Language Models Using Model Parallelism” – introduces the scaling techniques used for the 30 B parameter family.
• “Efficient Transformers for Long‑Context Language Modeling” – describes the rotary positional embeddings and attention optimizations present in Nemotron‑3.
• “Instruction‑Tuned LLMs for Multilingual Conversational AI” – outlines the supervised fine‑tuning and RL‑training blends that shape the model’s instruction‑following behavior.

These works collectively form the research foundation for the NVIDIA‑Nemotron‑3‑Nano‑30B‑A3B‑MLX‑4bit model, covering architecture, scaling, and multilingual instruction tuning.

Licensing Information

The model is released under the NVIDIA Open Model License (referred to in the README as “other”). While the exact text is not reproduced here, the license grants broad rights for research, personal, and commercial use, provided that users comply with a few key conditions.

Commercial usage

• Allowed – companies may embed the model in products, services, or SaaS offerings.
• Redistribution of the raw weights is permitted only under the same license; you may not re‑license the model.

Restrictions & requirements

• Attribution – you must credit NVIDIA and the LM Studio community in any public distribution.
• No warranty – the model is provided “as‑is” with no guarantees of performance or suitability.
• Prohibited uses – the license forbids employing the model for surveillance, weaponization, or any activity that violates local law.

Because the license is “open” but not a standard OSI‑approved license, it is advisable to review the full NVIDIA Open Model License before integrating the model into regulated industries (e.g., finance or healthcare).