SmolLM2-135M-Instruct

Technical Overview

SmolLM2‑135M‑Instruct is a compact, instruction‑tuned language model released by HuggingFaceTB. With 135 million parameters it belongs to the SmolLM2 family, a successor to SmolLM1 that focuses on on‑device efficiency while delivering strong performance on a broad set of natural‑language tasks. The model is built on a decoder‑only transformer architecture (similar to LLaMA) and has been trained on roughly 2 trillion tokens drawn from high‑quality web‑scale corpora such as FineWeb‑Edu, DCLM, The Stack, plus several curated datasets that will be released alongside the model.

Key capabilities include:

• Instruction following – the model can understand and respond to chat‑style prompts, system messages, and role‑based conversations.
• General‑purpose text generation – from short answers to longer essays, with controllable temperature and top‑p sampling.
• Zero‑shot reasoning – competitive scores on benchmarks that test commonsense, arithmetic, and knowledge retrieval.
• Function‑calling support (available in larger SmolLM2 variants) – the model can emit structured JSON for tool use.

Architecture highlights:

• Transformer decoder with 12 layers, 12 attention heads, and a hidden size of 768.
• LayerNorm and SwiGLU activation for a good balance between speed and expressiveness.
• Trained with a mixture of causal language modeling and supervised fine‑tuning (SFT) on instruction datasets, followed by Direct Preference Optimization (DPO) using the UltraFeedback corpus.

Intended use cases range from lightweight chat assistants on laptops or edge devices, to research prototypes that require fast iteration, to integration in web‑apps via transformers.js. Because the model fits comfortably in a single GPU’s memory, it is also a solid baseline for further fine‑tuning on domain‑specific data.

Benchmark Performance

SmolLM2‑135M‑Instruct was evaluated with the lighteval suite in a zero‑shot setting. The most relevant metrics for a 135 M‑parameter instruction model are:

• IFEval (average prompt/inst): 29.9 % – a notable jump over the non‑instruction baseline (17.2 %).
• MT‑Bench: 19.8 % – reflects improved multi‑turn conversational quality.
• HellaSwag: 40.9 % – better than the base model (38.9 %).
• ARC (average): 37.3 % – demonstrates stronger reasoning on multiple‑choice science questions.
• PIQA: 66.3 % – solid commonsense reasoning.
• MMLU (cloze): 31.5 % – competitive for a model of this size.

These benchmarks matter because they test a model’s ability to understand instructions, perform reasoning, and retrieve factual knowledge without task‑specific fine‑tuning. Compared to the original SmolLM1‑135M and other 100‑150 M‑parameter models (e.g., TinyLlama‑Chat), SmolLM2‑135M‑Instruct consistently outperforms on instruction‑following metrics while staying within the same memory footprint.

Hardware Requirements

Inference with SmolLM2‑135M‑Instruct is intentionally lightweight. The model can be loaded in:

• VRAM: ~2 GB (FP16) or ~1 GB (int8 quantized) for the raw model. Adding the tokenizer and a modest batch size stays comfortably under 4 GB.
• Recommended GPU: Any recent consumer GPU with at least 4 GB of VRAM (e.g., RTX 3060, RTX 2070 Super) for low‑latency chat. For higher throughput, a GPU with 8 GB+ (RTX 3080, A100) allows larger batch sizes.
• CPU inference: Viable on modern CPUs (8‑core + AVX‑512) but expect 3‑5× slower generation compared to GPU.
• Storage: The model checkpoint (safetensors) occupies ~1 GB. Including the tokenizer and auxiliary files, allocate ~2 GB of disk space.
• Performance: On a RTX 3060, typical generation speed is ~30 tokens / second with temperature=0.2 and top_p=0.9. The accelerate library can automatically shard the model across multiple GPUs for faster batch processing.

Use Cases

Because SmolLM2‑135M‑Instruct balances size and instruction ability, it shines in scenarios where resources are limited but conversational quality matters:

• On‑device assistants: Personal chatbots on laptops, smartphones, or IoT devices without internet connectivity.
• Customer‑service automation: Lightweight agents that can handle FAQs, ticket triage, or simple troubleshooting.
• Educational tools: Interactive tutoring that can rewrite text, summarize passages, or generate practice questions.
• Research prototyping: Fast iteration on prompt engineering, tool‑use experiments, or low‑resource fine‑tuning.
• Web‑app integration: Using transformers.js to run the model directly in browsers for privacy‑preserving chat.

Training Details

Training proceeded in three stages:

1. Base pre‑training: 2 trillion tokens from a mix of FineWeb‑Edu, DCLM, The Stack, and proprietary filtered corpora. The objective was standard causal language modeling.
2. Supervised fine‑tuning (SFT): Public instruction datasets plus a curated “smol‑talk” dataset (available here) were used to teach the model to follow chat‑style prompts.
3. Direct Preference Optimization (DPO): The model was further refined using the UltraFeedback binarized dataset, aligning outputs with human preferences without reinforcement learning.

Training compute was performed on a cluster of NVIDIA A100 GPUs (40 GB VRAM) for several weeks, employing mixed‑precision (FP16) and gradient checkpointing to fit the 135 M‑parameter model in memory. The resulting checkpoint is offered in safetensors format for fast loading, and an ONNX export is also available for inference on alternative runtimes.

Related Papers

The model’s design and evaluation are described in the following pre‑prints:

• SmolLM2: Compact, Instruction‑Tuned Language Models – the primary paper introducing the family, covering architecture, training data, and benchmark results.
• UltraFeedback (used for DPO) – see the UltraFeedback dataset repository for details on preference‑based fine‑tuning.

These works provide the theoretical foundation for the model’s token‑level training, instruction fine‑tuning pipeline, and the Direct Preference Optimization method that improves alignment with human preferences.

Licensing Information

The README lists the model under the Apache‑2.0 license, while the model card’s “license” field is marked “unknown”. In practice, the Apache‑2.0 terms apply to the released checkpoint and associated code. This permissive license:

• Allows commercial use, redistribution, and modification.
• Requires that a copy of the license be included with any distribution.
• Mandates attribution to the original authors (HuggingFaceTB) and a notice of any changes.
• Provides an explicit patent‑grant, protecting downstream users from patent litigation.

If you plan to embed the model in a product, ensure you keep the license file and credit the authors as stipulated. There are no additional usage restrictions such as “non‑commercial only”.