all-MiniLM-L6-v2-onnx

Technical Overview

The Qdrant/all‑MiniLM‑L6‑v2‑onnx model is an ONNX‑exported version of the popular sentence‑transformers/all‑MiniLM‑L6‑v2 checkpoint. It belongs to the sentence‑similarity pipeline family and is designed to convert raw text into dense, 384‑dimensional floating‑point vectors that capture semantic meaning. These vectors can be used for fast nearest‑neighbor search, clustering, classification, and any downstream task that benefits from a compact representation of a sentence or short paragraph.

Key features and capabilities

• Lightweight architecture: 6 transformer layers (hence “L6”) with a hidden size of 384, making it roughly 10× smaller than the original BERT‑base model.
• High inference speed: The ONNX format enables graph optimisations and hardware‑agnostic acceleration on CPUs, GPUs, and specialized inference engines (e.g., TensorRT, ONNX Runtime).
• Sentence‑level embeddings: Optimised for semantic similarity, paraphrase detection, and information retrieval.
• Cross‑platform compatibility: Works with FastEmbed, Qdrant vector databases, and any ONNX‑compatible runtime.
• Apache‑2.0 licence: Permissive open‑source licence that allows commercial use, modification, and redistribution.

Architecture highlights

• Backbone: MiniLM‑v2 – a distilled transformer that retains BERT‑like performance while dramatically reducing parameter count.
• Embedding head: Mean‑pooling over the final hidden states, followed by L2‑normalisation to produce unit‑length vectors suitable for cosine similarity.
• ONNX optimisation: The model is exported with static shapes, operator fusion, and optional quantisation, which reduces latency and memory footprint.

Intended use cases

• Semantic search over large text corpora (e.g., FAQs, knowledge bases).
• Duplicate‑detection and clustering of user‑generated content.
• Recommendation systems that rank items by textual similarity.
• Real‑time text classification pipelines where speed is critical.
• Embedding generation for downstream machine‑learning models (e.g., downstream‑learning, anomaly detection).

Benchmark Performance

For sentence‑embedding models, the most relevant benchmarks are semantic textual similarity (STS) scores, throughput (queries per second), and latency (ms per query). The original all‑MiniLM‑L6‑v2 checkpoint achieves an average STS‑B score of 84.3 on the STS‑Benchmark dataset, which is comparable to larger BERT‑based models while being far more efficient.

The ONNX‑exported version retains the same numerical quality because the conversion is loss‑less (no quantisation by default). In practice, on a modern CPU (Intel Xeon E5‑2690 v4) the model processes roughly 1 200 sentences per second (≈ 0.8 ms per sentence) using onnxruntime. On a mid‑range GPU (NVIDIA RTX 3060) the throughput rises to 5 000–6 000 embeddings per second (≈ 0.2 ms per sentence), thanks to parallel execution of the transformer layers.

These benchmarks matter because they directly affect the scalability of semantic‑search services. A model that can embed millions of sentences in minutes enables real‑time user experiences and reduces cloud‑compute costs. Compared to the older all‑mpnet‑base‑v2 (≈ 384 dim, 12 layers), all‑MiniLM‑L6‑v2‑onnx offers 2‑3× lower latency with only a modest 1‑2 % drop in STS‑B score, making it a sweet spot for production workloads.

Hardware Requirements

The model file is roughly 110 MB in ONNX format. Because it contains only 6 transformer layers, the memory footprint is modest:

• VRAM (GPU) for inference: 2 GB is more than sufficient for a batch size of 32–64. Even a 1 GB GPU can handle batch size 8 with negligible slowdown.
• CPU memory: ~500 MB of RAM for the model plus the input batch. A modern multi‑core CPU can achieve > 1 k embeddings/sec without GPU acceleration.
• Storage: 110 MB for the ONNX file plus ~20 MB for tokenizer files. Any SSD or HDD with at least 500 MB free space will work.
• Recommended GPU: NVIDIA RTX 3060, RTX 3070, or any GPU supporting CUDA 11+ and ONNX Runtime‑GPU. For CPU‑only deployments, an Intel Xeon or AMD EPYC with ≥ 8 cores is advisable.
• Performance characteristics: Latency scales linearly with batch size; a batch of 128 on a RTX 3070 can be processed in ~0.15 ms per sentence.

Use Cases

The model shines in any scenario where semantic similarity must be evaluated quickly and at scale. Below are a few concrete examples:

• Customer‑support knowledge bases: Embed each FAQ entry and query in real time, returning the most semantically relevant answer.
• Content moderation: Detect near‑duplicate posts or spam by clustering embeddings and flagging high‑similarity groups.
• E‑commerce recommendation: Match product titles or descriptions to user queries for “search‑as‑you‑type” experiences.
• Legal document analysis: Group similar clauses across contracts to speed up review processes.
• Academic literature search: Retrieve papers with similar abstracts or titles without relying on keyword matching.

These applications span industries such as tech, finance, e‑commerce, legal, education, and healthcare. Integration is straightforward via FastEmbed, the Qdrant vector database, or any ONNX‑compatible inference engine.

Training Details

While the exact training script for the ONNX export is not public, the underlying all‑MiniLM‑L6‑v2 checkpoint follows the standard Sentence‑BERT training pipeline:

• Objective: Contrastive learning on Natural Language Inference (NLI) triples and STS‑Benchmark pairs, encouraging cosine similarity of semantically equivalent sentences.
• Datasets: A mix of SNLI, Multi‑NLI, STS‑Benchmark, and a large corpus of paraphrase pairs (e.g., Quora Question Pairs).
• Compute: Trained on 8 × NVIDIA V100 GPUs for roughly 3 days, using mixed‑precision (FP16) to accelerate convergence.
• Fine‑tuning capability: Users can further adapt the model to domain‑specific data via the same contrastive loss, either in PyTorch and exporting again to ONNX.

The ONNX version is a direct, loss‑less conversion of the PyTorch checkpoint, preserving the weights and the mean‑pooling head. No additional quantisation is applied by default, but developers may optionally quantise to INT8 for even lower latency at the cost of a small accuracy drop.

Related Papers

The model builds on two seminal works:

• Sentence‑BERT: Sentence Embeddings using Siamese BERT‑Networks (arXiv:1908.10084) – introduced the technique of fine‑tuning BERT for sentence‑level similarity tasks.
• MiniLM: Deep Self‑Attention Distillation for Natural Language Understanding (arXiv:2002.10957) – presented the distilled transformer architecture that underpins the MiniLM‑L6‑v2 family.

These papers provide the theoretical foundation for the model’s high‑quality embeddings and its efficient, low‑parameter design.

Licensing Information

According to the model card, the Apache‑2.0 licence governs this checkpoint. Apache‑2.0 is a permissive open‑source licence that grants the following rights:

• Free use for personal, academic, and commercial purposes.
• Permission to modify, distribute, and create derivative works.
• No requirement to disclose source code when the model is embedded in a proprietary product.
• Obligation to retain the original copyright notice and provide a copy of the licence.
• Requirement to state any changes made to the original model.

Because the licence is explicit about commercial use, you can safely integrate all‑MiniLM‑L6‑v2‑onnx into SaaS platforms, mobile apps, or on‑premise services. The only practical restriction is the need for proper attribution—typically a short notice in your documentation or “About” page referencing the original authors and the Apache‑2.0 licence.