Llama-3.2-1B-Instruct-FP8

Technical Overview

The Llama‑3.2‑1B‑Instruct‑FP8 model is a quantized, instruction‑tuned version of Meta’s Llama‑3.2‑1B‑Instruct. It retains the original Llama‑3 architecture (a decoder‑only Transformer with 1 billion parameters) while applying aggressive 8‑bit floating‑point (FP8) quantization to both weights and activations. The result is a model that behaves like a small‑scale assistant‑style LLM but runs with roughly half the GPU memory footprint and up to twice the matrix‑multiply throughput of the unquantized 16‑bit counterpart.

Key features and capabilities include:

• FP8 activation and weight quantization (symmetric static per‑channel for weights, per‑tensor for activations).
• ≈50 % reduction in VRAM usage and disk size.
• 2× higher inference throughput on compatible GPUs (e.g., NVIDIA A100, RTX 4090).
• Multilingual instruction following in English, German, French, Italian, Portuguese, Hindi, Spanish and Thai.
• Optimized for vLLM and OpenAI‑compatible serving.

Architecture highlights:

• Base model: meta‑llama/Llama‑3.2‑1B‑Instruct (decoder‑only Transformer, 32 layers, 2 k hidden size, 32 k context window).
• Quantization: FP8 (E4M3) applied to every Linear layer inside transformer blocks; the final language‑model head (lm_head) is left in higher precision to preserve output fidelity.
• Calibration: 512 representative sequences from Neural Magic’s LLM‑compression calibration dataset, processed with the llm‑compressor library.

Intended use cases are assistant‑style chat, code assistance, and general‑purpose text generation in the eight supported languages. The model is marketed for both commercial products and research experiments that require a low‑memory footprint without sacrificing the quality of a 1 B‑parameter LLM.

Benchmark Performance

The most relevant benchmarks for a 1 B‑parameter instruction‑tuned model are MMLU, ARC‑Challenge, GSM‑8K, Hellaswag, Winogrande and TruthfulQA. These evaluate a mix of knowledge, reasoning, commonsense, and factual correctness—key traits for an assistant‑type LLM.

According to the README, the FP8‑quantized Llama‑3.2‑1B‑Instruct‑FP8 achieves scores within 1 % of the unquantized model across all listed benchmarks. This tiny accuracy loss is achieved while cutting memory usage in half and doubling inference speed.

Why these benchmarks matter:

• MMLU – measures broad academic knowledge across 57 subjects.
• ARC‑Challenge – tests advanced reasoning on multiple‑choice science questions.
• GSM‑8K – evaluates quantitative reasoning and math problem solving.
• Hellaswag & Winogrande – probe commonsense and contextual understanding.
• TruthfulQA – gauges the model’s ability to generate factually correct answers.

Compared with other 1 B‑parameter models (e.g., Mistral‑7B‑v0.1‑quantized, Gemma‑2B), Llama‑3.2‑1B‑Instruct‑FP8 offers a superior balance of multilingual instruction following and near‑full‑precision accuracy, making it a strong candidate for low‑cost, high‑throughput deployments.

Hardware Requirements

VRAM for inference: The FP8 quantization halves the memory footprint of the original 1 B model. In practice, a single GPU with 6 GB of VRAM can load the model comfortably when using the vLLM backend with tensor parallelism set to 1. For larger batch sizes or longer context windows (up to 8192 tokens), a 8 GB‑12 GB GPU is recommended.

Recommended GPU specifications:

• NVIDIA RTX 4090 (24 GB) – over‑kill but enables high‑throughput batch inference.
• NVIDIA A100 40 GB – ideal for multi‑GPU tensor‑parallel deployments.
• Any GPU supporting FP8 (or at least FP16 emulation) and CUDA 12+.

CPU requirements: The model is lightweight enough that a modern 8‑core CPU (e.g., AMD Ryzen 7 5800X or Intel i7‑12700K) can handle tokenization and I/O without becoming a bottleneck. For production serving, a CPU with ≥16 GB RAM is advisable.

Storage needs: The FP8‑quantized checkpoint is roughly 2 GB (safetensors format). Adding the tokenizer and configuration files brings the total to ≈2.5 GB. SSD storage is recommended for fast loading.

Performance characteristics: On a single RTX 4090, the model can generate ~200 tokens/second with the default sampling parameters (temperature 0.6, top‑p 0.9). Using vLLM’s OpenAI‑compatible server can scale linearly with additional GPUs.

Use Cases

Because the model is small, multilingual, and quantized for speed, it shines in scenarios where low latency and low hardware cost are paramount.

• Customer‑service chatbots – multilingual assistants that can run on a single GPU in a SaaS environment.
• Embedded AI for edge devices – the 2 GB footprint allows deployment on high‑end edge servers or on‑premise workstations.
• Educational tutoring – supports English, German, French, Italian, Portuguese, Hindi, Spanish and Thai, making it suitable for language‑learning platforms.
• Rapid prototyping & research – researchers can experiment with instruction‑tuned LLMs without needing multi‑GPU clusters.
• Content moderation & QA – the model’s near‑full‑precision accuracy on TruthfulQA makes it a good candidate for fact‑checking pipelines.

Integration is straightforward via the vLLM library, which also offers an OpenAI‑compatible endpoint, allowing the model to be used with existing LangChain, LlamaIndex, or custom REST wrappers.

Training Details

The base model meta‑llama/Llama‑3.2‑1B‑Instruct was originally trained on a mixture of publicly available text corpora (the “Mixture‑of‑Datasets” approach) for a total of ~1 trillion tokens, using a standard causal‑language‑model objective with instruction‑following fine‑tuning on a curated set of prompts and responses.

For the FP8 version, the authors performed post‑training quantization only—no additional pre‑training or instruction‑tuning. The quantization recipe was applied with the llm‑compressor library:

• Calibration dataset: 512 samples from the Neural Magic LLM compression calibration dataset.
• Quantization scheme: symmetric static per‑channel for linear weights, per‑tensor for activations, FP8 (E4M3) format.
• Tooling: QuantizationModifier and oneshot calibration routine.

The resulting model retains the original instruction‑following abilities while fitting into half the memory budget. Because the quantization is post‑training, the model can be fine‑tuned further (e.g., LoRA, QLoRA) on downstream tasks without needing to de‑quantize the weights.

Related Papers

The Llama‑3 series is built on research described in Meta’s Llama 3 technical report and the LLaMA: Open and Efficient Foundation Language Models paper. The FP8 quantization technique follows the work of “FP8 Training: 8‑bit Floating Point for Deep Learning” and the Neural Magic llm‑compressor framework, which is based on the quantization research presented in “Post‑Training Quantization for Large Language Models”.

These papers collectively inform the model’s architecture (decoder‑only Transformer), its instruction‑tuning methodology, and the aggressive FP8 quantization that enables the dramatic speed‑memory trade‑off.

Licensing Information

The model is released under the Llama3.2 license, which is a derivative of Meta’s Llama 3.2 licensing terms. While the README lists the license as “unknown”, the official model card indicates the Llama3.2 license, which typically permits non‑commercial research and commercial use with attribution provided that the user complies with Meta’s usage policy (no illegal content, no weaponization, respecting trade‑compliance laws).

Commercial usage: The license does not prohibit commercial deployment, but you must:

• Include a clear attribution to Meta and RedHatAI/Neural Magic.
• Ensure that the downstream product does not violate applicable laws (e.g., export controls, sanctions).
• Follow any model‑specific content‑policy restrictions (no disallowed content, no use for illegal activities).

Restrictions & requirements:

• No redistribution of the model weights under a different license.
• If you modify the model, you must label it as a derivative and retain the original license text.
• Some jurisdictions may require additional compliance checks for AI models (e.g., EU AI Act).

Attribution: A typical attribution line could be: “Llama‑3.2‑1B‑Instruct‑FP8 © Meta LLM, RedHatAI, Neural Magic, used under the Llama3.2 license.”