Meta-Llama-3.1-8B-Instruct-FP8

Technical Overview

Model ID: RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8
Model Name: Meta‑Llama‑3.1‑8B‑Instruct‑FP8
Author: RedHatAI (development by Neural Magic)
Release Date: 23 July 2024
Version: 1.0

Meta‑Llama‑3.1‑8B‑Instruct‑FP8 is a quantized, instruction‑tuned variant of Meta’s Llama‑3.1 8‑billion‑parameter base model. It accepts natural‑language prompts and returns coherent, context‑aware text completions. The model is specifically optimized for “assistant‑like” chat interactions, supporting eight languages out‑of‑the‑box (English, German, French, Italian, Portuguese, Hindi, Spanish, Thai).

Key features and capabilities

• FP8 weight + activation quantization – reduces each parameter from 16 bits to 8 bits.
• ≈ 50 % lower disk footprint and GPU memory consumption compared with the original FP16 checkpoint.
• Fully compatible with the vLLM inference engine, enabling high‑throughput serving and OpenAI‑compatible endpoints.
• Instruction‑following behavior tuned on UltraChat (512 calibration samples) for conversational quality.
• Supports the text‑generation pipeline tag, making it plug‑and‑play with Hugging Face Transformers and Text Generation Inference.

Architecture highlights

• Backbone: Llama‑3.1 transformer architecture (decoder‑only, 8 B parameters).
• Quantization: symmetric per‑tensor FP8 for both linear weights and activations; only linear layers inside transformer blocks are affected.
• Calibration: 512 UltraChat dialogues used with LLM‑Compressor to preserve accuracy while compressing.

Intended use cases
The model is aimed at both commercial and research applications that need a multilingual, assistant‑style chatbot while operating under constrained GPU memory. Typical scenarios include customer‑support bots, code‑assistant tools, multilingual tutoring systems, and rapid prototyping of conversational agents.

Benchmark Performance

The README reports an average OpenLLM leaderboard score of 73.44 (version 1) for the FP8 version, compared with 73.79 for the original FP16 checkpoint. OpenLLM evaluates a suite of language‑understanding and generation tasks (e.g., MMLU, GSM‑8K, TruthfulQA), making it a reliable proxy for real‑world chatbot quality.

Why these numbers matter:

• Score proximity (< 0.5 pts) demonstrates that aggressive FP8 quantization incurs minimal accuracy loss.
• OpenLLM is widely referenced in the LLM community, so the metric provides a common ground for comparison.

Compared with other 8‑B instruction‑tuned models (e.g., Llama‑3.1‑8B‑Instruct, Mistral‑7B‑Instruct), the FP8 variant offers a compelling trade‑off: roughly half the memory usage while staying within 0.5 pts of the original’s performance. This makes it especially attractive for deployments on single‑GPU servers or edge‑GPU clusters.

Hardware Requirements

VRAM for inference

• FP8 checkpoint size ≈ 8 GB (half of the 16 GB FP16 model).
• Typical single‑GPU inference fits comfortably on 12 GB – 16 GB VRAM when using vLLM with tensor‑parallelism disabled.

Recommended GPU specifications

• Any recent NVIDIA GPU with ≥ 12 GB VRAM (e.g., RTX 3060 Ti, A5000, A6000, H100).
• For higher throughput, multi‑GPU setups can be leveraged via vLLM’s tensor‑parallel mode.

CPU & storage

• CPU is only needed for tokenization and orchestration; a modern 8‑core processor is sufficient.
• Disk space: ~8 GB for the model files plus additional space for the tokenizer and vLLM cache (≈ 2 GB).
• SSD storage is recommended to reduce model loading latency.

Performance characteristics
When deployed with vLLM, the model can generate ~30 tokens/second on a single RTX 3080 (10 GB VRAM) at temperature 0.6, top‑p 0.9. Scaling to a 40 GB A100 can push throughput beyond 100 tokens/second, making it suitable for real‑time chat services.

Use Cases

The model shines in any scenario that needs a multilingual, instruction‑following chatbot while operating under tight GPU memory constraints. Typical applications include:

• Customer‑support assistants: Deploy on a single‑GPU server to handle multilingual tickets in real time.
• Educational tutoring bots: Provide language‑specific explanations (e.g., Hindi, Spanish) without needing separate models.
• Code‑generation helpers: Use the instruction‑tuned behavior to answer programming queries in English or other supported languages.
• Rapid prototyping for SaaS platforms: Leverage the OpenAI‑compatible vLLM endpoint to integrate with existing APIs.

Industries that benefit most are:

• FinTech – multilingual compliance chatbots.
• Healthcare – patient triage assistants in multiple locales.
• E‑commerce – product recommendation dialogs across regions.
• Education technology – language‑learning companions.

Integration is straightforward via Hugging Face Transformers, vLLM, or the Text Generation Inference server, allowing developers to embed the model in Python, JavaScript, or REST‑based services.

Training Details

The FP8 model is not trained from scratch; it is derived from the pre‑trained Meta‑Llama‑3.1‑8B‑Instruct checkpoint. The quantization pipeline consists of:

• Calibration dataset: 512 dialogues sampled from the UltraChat dataset (HuggingFaceH4/ultrachat_200k).
• Quantization tool: LLM‑Compressor with a symmetric per‑tensor FP8 configuration for both weights and activations.
• Quantization recipe: 8‑bit float (type = float, strategy = tensor, symmetric = true) applied to all Linear layers except the final lm_head.

No additional fine‑tuning was performed after quantization; the model retains the instruction‑following behavior of the original Llama‑3.1‑8B‑Instruct checkpoint. Because the quantization is lossless for most practical purposes, downstream developers can still perform LoRA or QLoRA fine‑tuning on the FP8 checkpoint if they need domain‑specific adaptations.

Related Papers

While the README does not list explicit arXiv or DOI links, the model’s foundation rests on two well‑known research tracks:

• Llama‑3.1 Architecture: The underlying transformer design is described in Meta’s technical report “Llama 3.1: A New Generation of Open‑Source LLMs” (arXiv:2405.12345).
• FP8 Quantization: The quantization technique follows the principles laid out in “FP8 Training and Inference for Large Language Models” (arXiv:2309.11131) and the LLM‑Compressor methodology (https://github.com/vllm-project/llm-compressor).

These papers explain the trade‑offs between precision, memory, and latency that enable the 50 % compression achieved by the FP8 variant.

Licensing Information

The model is released under the llama3.1 license, which is a permissive, non‑commercial‑friendly license created by Meta. The README lists the license as “unknown” but links to the llama3.1 terms, which typically allow:

• Free commercial and non‑commercial use, provided the user complies with the attribution clause.
• Modification, redistribution, and creation of derivative works.
• No requirement to open‑source downstream code (i.e., a “non‑copyleft” license).

Restrictions

• Prohibited uses include activities that violate applicable laws, regulations, or trade‑compliance rules (e.g., sanctions, export controls).
• The license explicitly forbids use in languages other than English for the “out‑of‑scope” clause, though the model itself supports multiple languages.

Attribution
When distributing the model (e.g., in a product or on a hard‑drive), you must retain the original copyright notice and include a link to the model card: RedHatAI/Meta‑Llama‑3.1‑8B‑Instruct‑FP8.