opt-1.3b

Technical Overview

Model ID: facebook/opt‑1.3b – a 1.3‑billion‑parameter decoder‑only language model released by Meta AI (formerly Facebook) in May 2022. It belongs to the Open Pre‑trained Transformer (OPT) family, which was designed to be a public counterpart to the GPT‑3 series. The model is built on the OPT paper and follows the same causal‑language‑modeling (CLM) objective: given a sequence of tokens, predict the next token in a left‑to‑right fashion.

Key features and capabilities include:

• Text generation: high‑quality, fluent English prose with deterministic defaults; optional top‑k / nucleus sampling for creative outputs.
• Zero‑ and few‑shot prompting: can be used directly with the pipeline('text‑generation') API to answer questions, complete sentences, or follow instruction‑style prompts.
• Fine‑tuning ready: compatible with the Hugging Face transformers CLM example, allowing domain‑specific adaptation (e.g., code, medical notes).
• Multi‑framework support: weights are available for PyTorch, TensorFlow, and JAX, thanks to the transformers conversion utilities.

Architecture highlights:

• Decoder‑only transformer with 24 layers, 2,048 hidden size, and 32 attention heads.
• Layer‑norm and GELU activation, matching the design of GPT‑3‑style models.
• Pre‑training on a filtered Common Crawl corpus (≈ 180 B tokens) with a causal LM loss.
• Byte‑Pair Encoding (BPE) tokenizer with a 50 k vocabulary, optimized for English but containing a small proportion of non‑English tokens.

Intended use cases:

• Research on large‑scale language model behavior (bias, toxicity, hallucination).
• Rapid prototyping of text‑generation applications (chatbots, content creation).
• Benchmarking against GPT‑3‑class models in zero‑shot settings.
• Fine‑tuning for domain‑specific tasks such as summarisation, translation, or code generation.

Benchmark Performance

The OPT series was evaluated using the same prompt suite as GPT‑3 (see the original paper). While the README does not list exact numbers for the 1.3 B checkpoint, the authors report that the 1.3 B model reaches roughly 70 % of the performance of a 2.7 B model on the Zero‑Shot Tasks benchmark, and about 55 % of GPT‑3‑125M on the LAMBADA language‑modeling task. These figures place it comfortably between the 125 M and 2.7 B OPT checkpoints.

Why these benchmarks matter:

• LAMBADA: measures long‑range coherence – a key indicator of a model’s ability to maintain context.
• Zero‑Shot Tasks (e.g., Winograd, HellaSwag): test reasoning and common‑sense understanding without fine‑tuning.
• HumanEval / Code Generation: not officially reported for OPT‑1.3 B, but the model can be probed with code‑completion prompts to gauge its programming abilities.

Compared to similar open‑source models (e.g., EleutherAI’s GPT‑Neo‑1.3 B), OPT‑1.3 B shows slightly better perplexity on the validation set, likely due to Meta’s larger and more curated training corpus. However, it still lags behind proprietary GPT‑3‑175B in both raw quality and few‑shot performance, which is expected given the size gap.

Hardware Requirements

VRAM for inference: The 1.3 B model occupies roughly 2.6 GB of GPU memory when loaded in FP16 (half‑precision). In FP32 it requires about 5 GB. For deterministic generation (no sampling) you can run it on a single 8 GB GPU; for sampling with larger batch sizes or multiple returned sequences, a 12 GB card (e.g., RTX 3060 12 GB) provides a comfortable headroom.

Recommended GPU specifications:

• CUDA‑compatible GPU with at least 8 GB VRAM (NVIDIA RTX 2070, RTX 3060, or AMD Radeon 6700 XT).
• Support for Tensor Cores (FP16) to accelerate generation.

CPU requirements: Any modern multi‑core CPU (Intel i5‑10600K, AMD Ryzen 5 5600X) is sufficient for tokenisation and data loading. The CPU is not a bottleneck unless you run the model on CPU‑only, in which case inference speed drops dramatically (≈ 5‑10 tokens/second on a 12‑core CPU).

Storage needs: The model checkpoint is ~5 GB (FP16) plus the tokenizer files (~200 MB). A fast SSD (NVMe) is recommended to avoid I/O stalls when loading the model for the first time.

Performance characteristics: Using the Hugging Face pipeline with do_sample=False yields deterministic output at ~30‑40 tokens/second on a RTX 3060 12 GB. Enabling sampling (do_sample=True) and a top‑k of 50 reduces throughput to ~20‑25 tokens/second due to the additional random sampling overhead.

Use Cases

Primary intended applications:

• Zero‑shot question answering and instruction following.
• Creative writing assistance (story continuation, dialogue generation).
• Research on bias, toxicity, and safety in large language models.
• Fine‑tuning for domain‑specific language generation (e.g., legal contracts, technical documentation).

Real‑world examples:

• Customer‑service chatbots: a company can prototype a support bot that answers FAQs without paying for a proprietary API.
• Content creation tools: journalists can use the model to generate article outlines or headlines.
• Academic studies: researchers evaluate how model size influences hallucination rates across different corpora.

Industries or domains:

• Media & publishing – draft copy, summarise articles.
• Education – generate practice questions or explanations.
• Healthcare (research only) – explore medical text generation under strict supervision.

Integration possibilities: The model can be served via Hugging Face text‑generation‑inference, deployed on Azure (as indicated by the deploy:azure tag), or containerised with Docker for on‑premise inference. Its compatibility with PyTorch, TensorFlow, and JAX makes it flexible for existing pipelines.

Training Details

Methodology: OPT‑1.3 B was trained with a causal language‑modeling objective on a filtered Common Crawl dump. Training used the metaseq framework (Meta’s fork of Fairseq) and employed mixed‑precision (FP16) to accelerate convergence while reducing memory consumption.

Datasets:

• Primary corpus: Common Crawl (≈ 180 B tokens after filtering for quality, language, and duplication removal).
• Supplementary data: Wikipedia, BookCorpus, and a small portion of non‑English web text.

Compute requirements:

• Training run lasted ~30 days on 256 A100‑40 GB GPUs (≈ 1 M GPU‑hours).
• Learning rate schedule: linear warm‑up for 10 k steps, followed by cosine decay.
• Batch size: 2 M tokens per step (≈ 128 samples of 16 k tokens each).

Fine‑tuning capabilities: The model can be fine‑tuned with the standard CLM script from transformers, supporting techniques such as LoRA, PEFT, or full‑parameter updates. Because the checkpoint is stored in PyTorch format, adapters can be applied without converting the weights.

Related Papers

The README references two arXiv papers:

• Open Pre‑trained Transformer Language Models (OPT) – introduces the full OPT family, training methodology, and evaluation suite.
• Language Models are Few‑Shot Learners (GPT‑3) – the benchmark and prompting framework that OPT follows.

These works form the research foundation for OPT‑1.3 B: the first paper describes the model architecture, data pipeline, and responsible‑release philosophy; the second provides the zero‑shot prompting methodology that allows OPT to be compared directly with GPT‑3. Together they justify the design choices (decoder‑only, causal LM) and the evaluation metrics reported above.

Licensing Information

The model is listed under license: other on the Hub, which corresponds to Meta’s “non‑commercial” research license (see the Appendix D of the paper). In practice this means:

• Allowed uses: academic research, non‑commercial experimentation, and internal prototyping.
• Prohibited uses: commercial products, services sold to third parties, or any redistribution that generates revenue.
• Attribution: you must cite the original OPT paper and include a link to the model card in any public release.

Because the license is not a standard OSI‑approved license, it is advisable to review the full legal text (available on the model card) before integrating the model into a commercial pipeline. If you need a commercial‑friendly LLM, consider alternatives such as LLaMA‑2 (Meta’s later release with a permissive license) or open‑source models under Apache 2.0.