SmolLM2-135M

Technical Overview

What is this model? SmolLM2‑135M is the smallest member of the SmolLM2 family – a compact, instruction‑tuned causal language model with 135 million parameters. Built on the LLaMA‑style transformer architecture, it is designed to deliver strong zero‑shot and few‑shot performance while remaining lightweight enough for on‑device or low‑cost cloud inference.

Key features & capabilities

• English‑centric understanding and generation, with solid performance on reasoning, commonsense, and knowledge tasks.
• Instruction‑following abilities thanks to a supervised fine‑tuning (SFT) stage followed by Direct Preference Optimization (DPO) on the UltraFeedback dataset.
• Supports text rewriting, summarisation, and – for the larger 1.7 B variant – function‑calling. The 135 M model can still handle many prompt‑engineering patterns.
• Compatible with the transformers library, text‑generation pipelines, and can be deployed on Azure endpoints (as indicated by the tags).

Architecture highlights

• Transformer decoder with 24 layers, 32 attention heads, and a hidden dimension of 768 (typical of a 135 M LLaMA‑style model).
• Trained with a mixture of 2 trillion tokens drawn from diverse web‑scale corpora: FineWeb‑Edu, DCLM, The Stack, plus several newly filtered datasets that will be released separately.
• Model weights are stored in safetensors format for fast loading and memory‑mapping.
• Supports mixed‑precision (bfloat16) inference, reducing the memory footprint to roughly 720 MB.

Intended use cases

• On‑device assistants, chatbots, or edge‑AI applications where GPU memory is limited (e.g., smartphones, embedded PCs).
• Rapid prototyping of instruction‑following tasks such as summarisation, paraphrasing, or lightweight code generation.
• Research experiments that require a fast, low‑cost baseline for zero‑shot or few‑shot evaluation.

Benchmark Performance

SmolLM2‑135M is evaluated primarily on zero‑shot benchmarks that test language understanding, reasoning, and instruction following. The lighteval suite is used for consistency across models.

Base pre‑trained model

Instruction‑tuned variant

Why these benchmarks matter – HellaSwag, ARC, and MMLU test commonsense and factual reasoning; PIQA focuses on physical intuition; GSM8K evaluates arithmetic problem solving; BBH (Big‑Bench Hard) measures multi‑step reasoning. The instruction‑tuned scores (IFEval, MT‑Bench) directly reflect how well the model follows human‑written prompts, a critical metric for chatbot or assistant workloads.

Compared to the original SmolLM‑135M, SmolLM2‑135M shows consistent improvements across the board (e.g., +5.8 points on HellaSwag, +3.4 on ARC) and a sizable jump in instruction following (+12.7 on IFEval). This positions it competitively against other 100‑200 M‑parameter models such as Mistral‑7B‑tiny or TinyLlama‑1.1B‑Chat, while retaining a far smaller memory footprint.

Hardware Requirements

VRAM for inference – In bfloat16 mode the model occupies roughly 723 MB of GPU memory (as reported by model.get_memory_footprint()). With fp16 the footprint is slightly higher (~800 MB). Therefore a GPU with at least 2 GB VRAM can host the model comfortably, leaving headroom for the activation cache during generation.

Recommended GPU – Any modern NVIDIA GPU with 4 GB+ VRAM (e.g., RTX 3060, GTX 1660 Super, or the newer RTX 40‑series) provides smooth generation at batch size = 1. For multi‑GPU deployments, the accelerate library can automatically shard the model across GPUs via device_map="auto".

CPU considerations – The model can run on CPU‑only systems, but generation latency will be significantly higher (≈5‑10× slower). A recent multi‑core CPU (e.g., AMD Ryzen 7 5800X or Intel i7‑12700K) with ≥16 GB RAM is the minimum for acceptable performance.

Storage – The checkpoint (including tokenizer) is ~1.2 GB when stored as safetensors. SSD storage is recommended to keep load times under a few seconds.

Performance characteristics – On a single RTX 3060 (12 GB VRAM) the model can generate ~30 tokens per second in bfloat16, scaling linearly with batch size. The low memory footprint also enables deployment on edge devices such as NVIDIA Jetson or even high‑end smartphones via ONNX Runtime.

Use Cases

• Chatbot & virtual assistant – The instruction‑tuned variant can follow natural language prompts, making it suitable for lightweight conversational agents on mobile or web.
• Content rewriting & summarisation – Works well for paraphrasing articles, generating concise abstracts, or cleaning up user‑generated text.
• Educational tools – Can answer factual questions, explain concepts, or generate practice problems for language‑learning apps.
• Prototype RAG (Retrieval‑Augmented Generation) – Its small size allows it to be paired with a local vector store for fast, on‑device retrieval‑augmented pipelines.
• Research baseline – Provides a fast, low‑cost baseline for zero‑shot and few‑shot experiments in NLP research.

Training Details

Methodology – Training proceeded in two stages. First, a causal language model was trained on ~2 trillion tokens using a mixture of high‑quality web corpora. Second, an instruction‑tuned version was created via supervised fine‑tuning (SFT) on public instruction datasets plus a proprietary “smol‑talk” set, followed by Direct Preference Optimization (DPO) using the UltraFeedback dataset.

Datasets

• FineWeb‑Edu – a filtered educational subset of the FineWeb corpus.
• DCLM – a large collection of code‑related and documentation text.
• The Stack – a massive dump of open‑source code and technical documentation.
• Additional curated datasets (to be released) that focus on high‑quality instruction data.
• SFT dataset: smol‑smoltalk.
• Preference data: UltraFeedback.

Compute – The base model was trained on a cluster of NVIDIA A100 GPUs (40 GB) for several weeks, using mixed‑precision (bfloat16) training to keep the memory footprint manageable. The exact FLOP count is not disclosed, but a 2‑trillion‑token run at ~30 TFLOP/s per GPU is a reasonable estimate.

Fine‑tuning capabilities – Users can continue SFT with their own instruction data via the Hugging Face alignment‑handbook recipes (see the GitHub link in the README). The model also supports parameter‑efficient fine‑tuning methods such as LoRA or QLoRA, allowing adaptation with as little as 1 GB of GPU memory.

Related Papers

The model’s research foundation is described in the arXiv pre‑print “SmolLM2: Compact Language Models for On‑Device Use”. This paper details the token‑level training regime, the curated dataset mix, and the instruction‑tuning pipeline (SFT + DPO). The UltraFeedback dataset used for DPO is introduced in the UltraFeedback paper (not directly listed but referenced via the dataset link). These works collectively explain the performance gains over SmolLM1 and the design choices for a 135 M‑parameter model.

Licensing Information

The README lists the Apache‑2.0 license, while the model card tags show license: unknown. In practice, the Apache‑2.0 license is permissive: you may use, modify, and distribute the model (including for commercial purposes) provided you retain the copyright notice and include a copy of the license.

Commercial use – Allowed under Apache‑2.0. Companies can embed the model in SaaS products, on‑device applications, or internal tools without paying royalties. However, you must ensure that any downstream modifications also carry the same attribution.

Restrictions – The license does not grant rights to use any third‑party datasets that may be embedded in the training corpus if those datasets have conflicting terms. Users should verify that their intended use complies with the underlying data licenses (e.g., FineWeb‑Edu, DCLM, The Stack). Additionally, the model’s “unknown” tag suggests that some downstream platforms may require a manual check before redistribution.

Attribution – A typical attribution line would be: “SmolLM2‑135M © HuggingFaceTB, licensed under Apache‑2.0.” Include a link to the model card and the Apache‑2.0 license text.