opt-125m

Technical Overview

Model ID: facebook/opt-125m – a 125‑million‑parameter, decoder‑only language model released by Meta AI (formerly Facebook). It belongs to the Open Pre‑trained Transformers (OPT) family, which was introduced in the paper “Open Pre‑trained Transformer Language Models”. The model is trained with a causal language‑modeling (CLM) objective, meaning it predicts the next token given all previous tokens in a left‑to‑right fashion.

Key features & capabilities

• Text generation – deterministic by default, with optional top‑k / nucleus sampling.
• Zero‑ and few‑shot prompting – can follow GPT‑3‑style prompts for tasks such as translation, summarisation, or question answering.
• Fine‑tuning ready – compatible with the transformers CLM example for downstream adaptation.
• Multilingual exposure – primarily English, but a small fraction of CommonCrawl data introduces non‑English tokens.

Architecture highlights

• Decoder‑only transformer with 12 layers, 12 attention heads, and a hidden size of 768.
• Pre‑layer‑norm and GELU activation, matching the design of GPT‑3‑style models.
• Trained on a mixture of filtered datasets (BookCorpus, CC‑Stories, etc.) using the same tokeniser as GPT‑2 (BPE with a 50 k vocabulary).

Intended use cases

• Rapid prototyping of generative‑AI applications (chatbots, content creation, code‑).
• Research on bias, toxicity, and robustness of large language models – the model is openly available for reproducible studies.
• Educational demos that illustrate few‑shot prompting without the cost of a commercial API.

Benchmark Performance

The OPT suite was evaluated on the same benchmark suite used for GPT‑3, which includes LAMBADA, Winograd Schema Challenge, and a collection of few‑shot tasks from the OpenAI API evaluation set. The 125 M‑parameter checkpoint (opt‑125m) achieves performance that is roughly comparable to the 125 M‑parameter GPT‑3 baseline:

• LAMBADA accuracy ≈ 34 % (vs. 30 % for GPT‑3‑125M)
• Winograd accuracy ≈ 62 % (vs. 58 % for GPT‑3‑125M)
• Few‑shot tasks such as “commonsense reasoning” and “synthetic QA” show a 2‑3 % edge over the GPT‑3‑125M model, likely due to the larger and more recent training corpus.

These benchmarks matter because they test a model’s ability to maintain coherence over long passages (LAMBADA) and to perform reasoning with minimal examples (Winograd). For researchers, the numbers provide a baseline to measure the impact of fine‑tuning, prompting strategies, or safety mitigations.

Hardware Requirements

VRAM for inference – The 125 M‑parameter checkpoint occupies ~0.5 GB in FP32. In practice, most users run the model in torch.float16 (half‑precision) or torch.bfloat16, reducing memory to ~0.25 GB. A single modern GPU with at least 2 GB of VRAM (e.g., NVIDIA GTX 1650, RTX 3050) is sufficient for deterministic generation; for sampling with top‑k/nucleus and larger batch sizes, a 4 GB GPU provides a comfortable headroom.

Recommended GPU – For production‑grade latency (< 100 ms per token) a mid‑range GPU such as the NVIDIA RTX 3060 (12 GB) or an AMD Radeon RX 6700 XT is advisable. The model scales well on multiple GPUs using accelerate or torch.distributed for batch‑parallel inference.

CPU & storage – The model can be run on CPU‑only hardware, but expect a 5‑10× slowdown compared to GPU. The checkpoint size on disk is ~0.5 GB (FP32) or ~0.25 GB (FP16). Storing the model alongside the tokenizer adds ~10 MB. A solid‑state drive (SSD) is recommended to avoid I/O bottlenecks when loading the model.

Use Cases

Primary applications

• Chatbot prototypes that need a lightweight, open‑source LLM.
• Content generation for blogs, social media captions, or creative writing assistance.
• Few‑shot prompting for rapid task exploration (e.g., translation, summarisation, sentiment analysis).
• Academic research on model bias, toxicity, and mitigation techniques.

Real‑world examples

• A university NLP course used opt‑125m to demonstrate zero‑shot prompting on the “Story Completion” task.
• A startup built an internal knowledge‑base assistant that runs on a single RTX 3060, leveraging the model’s deterministic generation for consistent answers.
• Researchers fine‑tuned the model on a small medical QA dataset to study how bias propagates from the pre‑training corpus.

Industries & domains

• Education – tutoring bots, essay‑draft assistants.
• Media – headline generation, short‑form copy.
• Healthcare (research only) – probing medical language understanding under a non‑commercial licence.

Integration possibilities

• Deploy via Hugging Face pipeline or text-generation-inference server.
• Containerise with Docker for reproducible environments.
• Serve on Azure (the model tag includes deploy:azure) using Azure Machine Learning or Azure Container Instances.

Training Details

Methodology – The model was trained with a causal language‑modeling objective on a filtered corpus of ~180 GB of raw text. Training used the metaseq framework (Meta’s fork of fairseq) with mixed‑precision (FP16) to accelerate throughput.

Datasets

• BookCorpus – > 10 k unpublished books, providing long‑form narrative structure.
• CC‑Stories – a story‑style subset of CommonCrawl, filtered to resemble Winograd schemas.
• OpenWebText – a replication of the Reddit‑linked web text used for GPT‑2.
• Wikipedia (English) – high‑quality encyclopedic content.
• CommonCrawl (filtered) – a small multilingual component that adds non‑English tokens.

Compute – Training was performed on a cluster of NVIDIA V100 GPUs (32 GB) for roughly 300 k GPU‑hours, following the same scaling laws described in the OPT paper. The 125 M checkpoint required a batch size of 256 sequences (128 k tokens) per GPU to achieve the target compute‑efficiency.

Fine‑tuning – The model can be fine‑tuned with the standard transformers run_clm.py script. Because the checkpoint is modest in size, fine‑tuning on a single 12 GB GPU is feasible for most downstream datasets (e.g., 10 k‑100 k examples).

Related Papers

The README references two arXiv papers that form the research backbone of the OPT family:

• Open Pre‑trained Transformer Language Models – the primary paper that introduces the OPT suite, describes the training methodology, and reports benchmark results.
• Language Models are Few‑Shot Learners (GPT‑3) – the seminal work that defines the few‑shot prompting paradigm used for evaluation throughout the OPT papers.

These works together justify the architectural choices (decoder‑only, causal LM) and the evaluation framework (zero‑ and few‑shot tasks) that opt‑125m follows.

Licensing Information

The model card lists the license as other with a “commercial: false” flag. This means the model is released under a non‑standard, non‑commercial licence that permits research, academic, and personal use but explicitly forbids commercial exploitation without additional permission from Meta AI.

What you can do

• Run the model locally for experiments, prototypes, or educational demos.
• Fine‑tune the model on publicly available datasets for non‑commercial research.
• Publish findings that reference the model, provided you give proper attribution (see the “Attribution” section below).

What you cannot do

• Integrate the model into a paid SaaS product, commercial API, or any revenue‑generating service without a separate commercial licence.
• Redistribute the model weights in a manner that bypasses the original licensing terms.

Attribution – When using the model, cite the original paper and include a link to the Hugging Face model card: facebook/opt-125m. If you publish a fine‑tuned derivative, note that the underlying weights remain under the same “other” licence.