distilbert-base-nli-mean-tokens

Technical Overview

Model ID: sentence-transformers/distilbert-base-nli-mean-tokens
Model Name: distilbert‑base‑nli‑mean‑tokens
Author: sentence‑transformers
License: Apache‑2.0 (as declared in the README)

This model is a sentence‑embedding model built on top of DistilBERT, a distilled version of BERT that retains ~97 % of BERT’s language understanding while being ~40 % faster and 60 % smaller. The model maps a single sentence or a short paragraph to a 768‑dimensional dense vector. Those vectors can be compared with cosine similarity, Euclidean distance, or other similarity measures to capture semantic relatedness.

Key features and capabilities

• Fast inference thanks to the lightweight DistilBERT backbone.
• Mean‑pooling over token embeddings (the “mean‑tokens” part) to produce a single fixed‑size sentence vector.
• Compatible with the Sentence‑Transformers library, the Hugging Face transformers API, ONNX, OpenVINO, and TensorFlow.
• Designed for downstream tasks such as semantic search, clustering, duplicate detection, and information retrieval.

Architecture highlights

• Transformer block: DistilBERT with a maximum sequence length of 128 tokens, 6 encoder layers, 12 attention heads, and a hidden size of 768.
• Pooling layer: Mean‑pooling over the token embeddings while respecting the attention mask (i.e., padding tokens are ignored). No CLS‑token or max‑pooling is used.
• Output dimension: 768‑dimensional sentence embedding.

Intended use cases

• Semantic similarity scoring between sentences or short paragraphs.
• Clustering large corpora of text (e.g., news articles, product reviews).
• Nearest‑neighbor search for question‑answering or recommendation engines.
• Pre‑processing step for downstream classification or regression tasks that benefit from dense text representations.

For a quick start, the model can be loaded with the SentenceTransformer class:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('sentence-transformers/distilbert-base-nli-mean-tokens')
embeddings = model.encode(["First sentence", "Second sentence"])

Benchmark Performance

Sentence‑embedding models are typically evaluated on semantic‑textual‑similarity (STS) benchmarks, paraphrase detection, and retrieval tasks such as the SBERT paper (STS‑B, STS‑12‑16). The distilbert‑base‑nli‑mean‑tokens model was originally fine‑tuned on the Natural Language Inference (NLI) datasets (SNLI + Multi‑NLI) and then evaluated on STS benchmarks.

• STS‑B average Pearson / Spearman: ~78 % (lower than newer SBERT models that reach >85 %).
• Mean‑pooling vs. CLS‑pooling: Mean‑pooling provides a modest boost in semantic similarity scores for short sentences.
• Inference speed: ~2 ms per sentence on a single RTX 3080 (FP16), roughly twice as fast as the original BERT‑base SBERT model.

These metrics matter because they directly reflect how well the embedding space preserves semantic relationships—a critical factor for search, clustering, and duplicate detection. While the model is deprecated in favor of higher‑quality SBERT variants, it still offers a useful baseline for low‑resource environments.

Hardware Requirements

VRAM for inference

• DistilBERT (768 dim, 6 layers) occupies ~300 MB of GPU memory when loaded in FP32.
• Using mixed‑precision (FP16) reduces the footprint to ~150 MB.

Recommended GPU specifications

• Any GPU with ≥ 4 GB VRAM (e.g., NVIDIA GTX 1650, RTX 2060) can run the model comfortably for batch sizes of ≤ 32.
• For high‑throughput workloads (batch ≥ 128), a GPU with ≥ 8 GB VRAM (RTX 3070, RTX 3080, A100) is advisable.

CPU requirements

• On CPU‑only inference, expect ~30–50 ms per sentence on a modern 8‑core processor (e.g., AMD Ryzen 7 5800X).
• Enabling ONNX Runtime or OpenVINO can cut CPU latency by ~30 %.

Storage needs

• Model files (weights, tokenizer, config) total ~350 MB.
• Additional space is required for the sentence‑transformers library (~150 MB) and any cached tokenizers.

Performance characteristics

• Mean‑pooling adds negligible overhead; the dominant cost is the transformer forward pass.
• Batching improves throughput linearly up to the GPU memory limit.

Use Cases

Primary intended applications

• Semantic similarity scoring for duplicate detection (e.g., detecting near‑duplicate support tickets).
• Fast semantic search over small‑to‑medium corpora (e.g., internal knowledge bases, product catalogs).
• Clustering of short texts for topic modeling or content recommendation.

Real‑world examples

• E‑commerce: Matching user queries to product titles to improve search relevance.
• Customer support: Grouping similar issue descriptions to suggest existing solutions.
• Academic research: Finding related abstracts in a literature database.

Industries or domains

• Retail & e‑commerce
• Finance (fraud detection via text similarity)
• Healthcare (clinical note clustering)
• Legal (case‑law similarity)

Integration possibilities

• Directly via the sentence‑transformers Python library.
• Exported to ONNX for deployment in C++ or Java services.
• Hosted on Hugging Face Inference Endpoints for low‑latency API access.

Training Details

The distilbert‑base‑nli‑mean‑tokens checkpoint was created by the sentence‑transformers team using the following methodology:

• Base architecture: DistilBERT (6 layers, 12 heads, 768 hidden units).
• Fine‑tuning dataset: A combination of the SNLI and Multi‑NLI corpora, providing sentence pairs with entailment, contradiction, and neutral labels.
• Training objective: Siamese contrastive loss (triplet loss) that pushes embeddings of semantically similar sentences together while separating dissimilar ones.
• Pooling strategy: Mean‑pooling over token embeddings, implemented as a separate Pooling module in the Sentence‑Transformers library.
• Compute resources: Trained on a single NVIDIA V100 GPU for ~12 hours, using a batch size of 32 and a learning rate of 2e‑5.
• Fine‑tuning capabilities: Users can further fine‑tune the model on domain‑specific NLI or paraphrase data by loading the checkpoint with SentenceTransformer and calling model.fit().

Related Papers

The model is built on the research presented in:

• Sentence‑BERT: Sentence Embeddings using Siamese BERT‑Networks (Reimers & Gurevych, 2019) – the foundational paper that introduced the SBERT framework and demonstrated the effectiveness of NLI‑fine‑tuned BERT models for semantic similarity.

DistilBERT itself is described in DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (Sanh et al., 2020), which explains the trade‑offs between model size and performance.

Licensing Information

The README declares an Apache‑2.0 license, which is a permissive open‑source license. The metadata field “license: unknown” is a discrepancy; the official model card and README confirm Apache‑2.0.

• Commercial use: Allowed. You may embed the model in commercial products, SaaS platforms, or APIs.
• Modification: You may modify the code, fine‑tune the weights, or re‑package the model.
• Distribution: You may redistribute the original or modified model, provided you retain the license notice and a copy of the Apache‑2.0 license.
• Patent grant: Apache‑2.0 includes an explicit patent license, reducing risk for commercial deployments.
• Attribution: Required. Cite the original SBERT paper (see the “Related Papers” section) and retain the license header in any redistributed files.

If you plan to use the model in a proprietary setting, ensure that any downstream dependencies (e.g., sentence‑transformers) are also compatible with Apache‑2.0.