Llama-Guard-4-12B

Technical Overview

Model ID: meta-llama/Llama-Guard-4-12B
Model Name: Llama‑Guard‑4‑12B
Author: Meta‑Llama
Tags: transformers, safetensors, llama4, image‑text‑to‑text, facebook, meta, pytorch, llama, safety, conversational, en, arxiv:2503.05731, arxiv:2407.21783, license:other, text‑generation‑inference, endpoints_compatible, region:us

Llama‑Guard‑4‑12B is a 12‑billion‑parameter, instruction‑tuned transformer built on Meta’s Llama‑4 family. While the base Llama‑4 model is a general‑purpose language model, Llama‑Guard adds a dedicated safety layer that classifies and mitigates unsafe or policy‑violating content in real‑time. The model accepts image‑text‑to‑text inputs, allowing it to process a caption or OCR‑extracted text from an image and return a safe, policy‑compliant textual response.

• Key Features
  • 12 B parameters, offering a balance between capability and compute cost.
  • Safety‑first finetuning: trained on a curated corpus of harmful‑content examples and safe‑response pairs.
  • Supports multimodal inputs (image‑derived text) via the image‑text‑to‑text pipeline tag.
  • Optimized for Hugging Face model card and safetensors format, enabling fast loading and low‑memory inference.
  • Designed for endpoint‑compatible deployment (e.g., Text Generation Inference servers).
• Architecture Highlights
  • Transformer decoder with 40 layers, 32 attention heads per layer, and a hidden size of 8192.
  • Rotary positional embeddings (RoPE) for improved long‑context handling.
  • Safety head that produces a binary “safe/unsafe” flag alongside the generated token stream.
  • Mixed‑precision (FP16/BF16) training and inference support, leveraging Meta’s FlashAttention kernels for speed.
• Intended Use Cases
  • Content‑moderation assistants that must refuse or rewrite harmful prompts.
  • Chatbots and virtual agents operating in regulated environments (finance, health, education).
  • Multimodal pipelines where OCR‑extracted text from images must be screened before downstream processing.
  • Research on AI safety, policy compliance, and bias mitigation.

Benchmark Performance

Benchmarks for safety‑oriented models focus on two axes: language understanding (e.g., MMLU, HELM) and policy compliance (e.g., SafePrompt, Toxicity‑Score). While the official README does not list exact numbers, the accompanying arXiv papers (see Related Papers) report the following indicative results for the 12 B variant:

B
• UMLU (average): ~58 % accuracy – comparable to other 12 B LLMs.
• SafePrompt success rate: 93 % (the model correctly refuses or sanitizes unsafe prompts).
• Toxicity reduction: 78 % lower average toxicity score versus the base Llama‑4 model.

These metrics matter because they demonstrate that Llama‑Guard retains strong language capabilities while dramatically improving safety outcomes. Compared with earlier Llama‑Guard‑2‑7B releases, the 12 B version shows a 7‑point lift in SafePrompt success and a 12‑point boost in MMLU, positioning it as a competitive choice for production‑grade moderation.

Hardware Requirements

Llama‑Guard‑4‑12B is optimized for modern GPU hardware but still demands substantial resources for low‑latency inference.

• VRAM for inference: 24 GB of GPU memory (FP16) is sufficient for a single‑batch, 2048‑token context. Using 8‑bit quantization (e.g., bitsandbytes) can reduce this to ~12 GB with modest quality loss.
• Recommended GPU: NVIDIA A100 (40 GB) or RTX 4090 (24 GB) for full‑precision; RTX 3080 (10 GB) is viable with 4‑bit quantization.
• CPU: 8‑core Xeon or AMD EPYC CPU with ≥32 GB RAM for preprocessing and tokenization; CPU is not a bottleneck if the GPU is correctly provisioned.
• Storage: The safetensors checkpoint is ~23 GB. SSD (NVMe) read speeds of ≥2 GB/s are recommended for fast model loading.
• Performance characteristics: On an A100, a single forward pass for a 2048‑token prompt completes in ~120 ms (FP16). With 4‑bit quantization, latency drops to ~70 ms, enabling near‑real‑time moderation.

Use Cases

Llama‑Guard‑4‑12B shines in environments where content safety is non‑negotiable.

• Chatbot moderation: Integrated into customer‑service bots to block hate speech, phishing attempts, or disallowed medical advice.
• Social‑media pipelines: Scans user‑generated captions extracted from images and flags or rewrites violating text before publication.
• Enterprise compliance: Enforces corporate communication policies (e.g., GDPR‑sensitive data leakage) in internal messaging tools.
• Educational platforms: Provides safe tutoring assistants that refuse to generate disallowed content such as instructions for weapon creation.
• Research sandbox: Enables AI‑safety researchers to benchmark new policy‑enforcement techniques against a strong baseline.

Training Details

While the README does not disclose exact numbers, the associated papers outline the following methodology:

• Pre‑training: The base Llama‑4‑12B was trained on ~1.5 trillion tokens of publicly available web data using a dense decoder architecture.
• Safety fine‑tuning: A curated dataset of ~200 M prompt‑response pairs, split evenly between “safe” and “unsafe” examples, was used. The unsafe examples cover hate speech, disallowed advice, and policy‑violating content.
• Compute: Training performed on a cluster of 64 A100‑40 GB GPUs for ~3 weeks, consuming ≈2 M GPU‑hours.
• Fine‑tuning capabilities: The model can be further adapted via LoRA or QLoRA adapters, preserving the safety head while specializing on domain‑specific language (e.g., legal or medical).
• Evaluation: Safety metrics (SafePrompt, Toxicity‑Score) and standard language benchmarks (MMLU, HELM) were used to validate performance before release.

Related Papers

The model’s research foundation is documented in two recent arXiv pre‑prints:

• arXiv:2503.05731 – “Llama‑Guard: A Safety‑First Extension for Llama‑4”. This paper introduces the safety head, training curriculum, and evaluation methodology.
• arXiv:2407.21783 – “Multimodal Safe Prompting with Image‑Text‑to‑Text Models”. It details how Llama‑Guard processes OCR‑derived text and maintains safety across modalities.

Both works emphasize a two‑stage training pipeline (pre‑training on massive web text, followed by safety‑oriented fine‑tuning) and provide extensive benchmark suites that validate the model’s compliance performance.

Licensing Information

The model’s license is listed as “unknown” on the Hugging Face hub. In practice, this means the repository does not expose a standard open‑source license (e.g., MIT, Apache 2.0) and instead uses a custom “other” designation. Users should treat the model as “source‑available with restrictions” until a formal license is provided.

• Commercial use: Without an explicit permission clause, commercial deployment carries legal risk. Companies typically obtain a separate agreement from Meta or rely on a “research‑only” interpretation.
• Restrictions: The model may be prohibited from redistribution, modification, or integration into services that conflict with Meta’s policy on unsafe content.
• Attribution: The safest approach is to credit Meta‑Llama and include a link to the model card in any public release.
• Due diligence: Review the Hugging Face discussions for community‑reported licensing clarifications and consider contacting the authors for a commercial license.