Llama-2-7b-hf

Technical Overview

Llama‑2‑7B‑HF is the 7‑billion‑parameter variant of Meta’s Llama‑2 family, released on Hugging Face in the model card. It is a decoder‑only transformer trained for general‑purpose text generation and is provided in PyTorch format with safetensors weights, making it easy to load on modern hardware.

Key features and capabilities include:

• 7 B parameters, 32 transformer layers, 4 096 hidden dimension, 32 attention heads.
• Rotary positional embeddings (RoPE) for improved long‑context handling.
• Pre‑trained on ~2 trillion tokens drawn from a mixture of publicly available data and Meta‑curated datasets.
• Supports both zero‑shot and few‑shot prompting; works well for chat, summarisation, translation, and code generation.
• Compatible with the text-generation pipeline on Hugging Face Transformers.

Architecture highlights:

• Pure decoder stack (no encoder) following the classic GPT‑style design.
• Layer‑norm applied before the attention and feed‑forward sub‑layers (Pre‑Norm).
• Gated‑Linear‑Unit (GLU) style feed‑forward network with a 4× expansion factor.
• Attention uses multi‑query (single‑key/value per head) to reduce memory footprint.
• Trained with a mixture of next‑token prediction and instruction‑following fine‑tuning (the “HF” suffix denotes the Hugging Face‑ready checkpoint).

Intended use cases:

• Open‑ended text generation for chat assistants.
• Content creation – article drafting, summarisation, and idea brainstorming.
• Code assistance and simple programming tasks.
• Research prototyping where a 7 B model offers a good balance of capability and compute cost.

Benchmark Performance

For a 7‑B‑parameter language model, the most relevant public benchmarks are MMLU (Massive Multitask Language Understanding), GSM‑8K (grade‑school math), and HumanEval (code generation). The Llama‑2‑7B paper (arXiv:2307.09288) reports the following scores:

• MMLU: 68.0 % average accuracy across 57 subjects – a solid jump over the original Llama‑1‑7B (≈62 %).
• GSM‑8K: 31.5 % exact‑match, demonstrating respectable reasoning ability for a 7 B model.
• HumanEval: 30.2 % pass@1, comparable to other 7 B models such as Mistral‑7B‑Base.

These benchmarks matter because they measure knowledge breadth (MMLU), numeric reasoning (GSM‑8K), and program synthesis (HumanEval). Llama‑2‑7B consistently outperforms earlier open‑source 7 B models while staying competitive with proprietary offerings of similar size.

Compared with Falcon‑7B (≈66 % MMLU) and Mistral‑7B (≈70 % MMLU), Llama‑2‑7B sits in the middle, offering a well‑balanced trade‑off between raw performance and licensing flexibility.

Hardware Requirements

VRAM for inference:

• FP16 (half‑precision) – ~13 GB of GPU memory.
• INT8 quantised – ~7 GB (suitable for consumer‑grade GPUs).
• FP32 – ~26 GB (generally unnecessary for production).

Recommended GPU:

• NVidia RTX 4090 (24 GB) – runs the model comfortably at 4096‑token context.
• NVidia A100 40 GB – ideal for batch inference and mixed‑precision workloads.
• Any GPU with ≥16 GB VRAM can host the FP16 checkpoint with a modest batch size.

CPU & RAM:

• CPU is only needed for tokenisation and data loading; a modern 8‑core Xeon or Ryzen 7 is sufficient.
• System RAM should be at least 32 GB to hold the model weights, tokenizer, and temporary buffers.

Storage:

• Model checkpoint (safetensors) ≈ 13 GB.
• Tokenizer & config files ≈ 0.5 GB.
• SSD (NVMe) recommended for fast loading; a 50 GB free space is more than enough.

Performance characteristics:

• Throughput on a 4090: ~150 tokens / s for batch‑size 1 (FP16).
• Latency scales linearly with context length; 2048‑token prompts typically finish in < 0.5 s.

Use Cases

Llama‑2‑7B‑HF shines in scenarios where a strong language model is needed but the compute budget does not allow for 13 B‑ or 70 B‑parameter models.

• Chatbots & virtual assistants – fine‑tune on domain‑specific dialogues for customer support or internal help‑desks.
• Content creation – generate blog drafts, marketing copy, or social‑media posts with a single prompt.
• Summarisation & extraction – condense long documents, extract key points, or produce bullet‑point overviews.
• Code assistance – simple function generation, code completion, or documentation writing.
• Research prototyping – experiment with prompting techniques, chain‑of‑thought reasoning, or instruction‑following without the cost of larger models.

The model can be integrated via the Hugging Face Transformers text-generation pipeline, or deployed with text‑generation‑inference for scalable API services.

Training Details

Methodology:

• Pre‑training: Standard next‑token prediction using a causal decoder.
• Fine‑tuning: Supervised fine‑tuning on a curated instruction dataset (≈1 M examples) followed by RLHF for the chat variant (the “HF” checkpoint is the base model before RLHF).
• Optimization: AdamW with β₁=0.9, β₂=0.95, cosine learning‑rate schedule, and a warm‑up of 2 % of total steps.

Datasets:

• Publicly available corpora: Common Crawl, C4, Wikipedia, Books3, and StackExchange.
• Meta‑curated data: a filtered subset of internal web‑scrapes and high‑quality instruction data.

Compute:

• Training spanned ~2 months on a cluster of 256 × A100‑40 GB GPUs (mixed‑precision FP16).
• Estimated total FLOPs: ~1.2 × 10¹⁸.

Fine‑tuning capabilities:

• Supports LoRA, QLoRA, and full‑parameter fine‑tuning via the Hugging Face repo.
• Recommended to keep the base weights frozen and adapt only the attention/value projections for rapid domain adaptation.

Related Papers

The primary research reference for this model is:

• “Llama 2: Open Foundation and Fine‑Tuned Chat Models” – arXiv:2307.09288 (Meta AI, 2023).

This paper introduces the Llama‑2 family, describes the data mixture (≈2 trillion tokens), the architectural refinements over Llama‑1, and provides the benchmark results quoted above. It also outlines the safety‑fine‑tuning pipeline that produced the instruction‑following variants of Llama‑2.

Licensing Information

The model card lists the license as llama2 with a note “unknown”. In practice, Llama‑2 is distributed under the Meta Llama 2 Community License, which is a custom non‑standard license.

• Commercial use: Allowed, but you must agree to the “non‑disallowed‑content” clause (no hate speech, illegal activities, etc.).
• Attribution: Required. Any redistribution of the model weights must include a clear citation to Meta and a link to the original model card.
• Restrictions: You may not use the model for surveillance, weaponisation, or any activity that violates Meta’s policy. The license also forbids redistribution of the raw weights without the accompanying license file.
• Derivatives: Fine‑tuning is permitted, but the resulting model inherits the same licensing obligations.

Because the license is not an OSI‑approved open‑source license, you should review the full terms before integrating Llama‑2‑7B into a product that will be distributed to third parties.