ms-marco-MiniLM-L2-v2

Technical Overview

Model ID: cross-encoder/ms-marco-MiniLM-L2-v2
Model Name: ms-marco-MiniLM-L2-v2
Author: cross‑encoder
Library: Sentence‑Transformers (also usable via 🤗 Transformers)
Pipeline Tag: text‑ranking

The cross‑encoder/ms‑marco‑MiniLM‑L2‑v2 is a lightweight, high‑throughput neural ranker built for the MS MARCO Passage Ranking task. Unlike bi‑encoders that embed queries and passages independently, a cross‑encoder concatenates the query and candidate passage, feeds the pair through a transformer, and directly predicts a relevance score. This “cross‑attention” design yields superior ranking quality at the cost of a modest increase in inference latency, making it ideal for re‑ranking a short list of candidates (e.g., the top‑k results from a BM25 or dense retrieval stage).

Key Features & Capabilities

• MiniLM backbone: The model uses the MiniLM‑L2 architecture (≈ 22 M parameters), offering a strong trade‑off between speed and accuracy.
• Sentence‑Transformers integration: One‑line inference with CrossEncoder and automatic handling of tokenisation, padding, and batching.
• Multi‑framework support: Available as a AutoModelForSequenceClassification in 🤗 Transformers, ONNX, OpenVINO, JAX, and safetensors, enabling deployment on CPUs, GPUs, and edge devices.
• English‑only: Trained on the English MS MARCO dataset, it excels at passage‑level relevance for English queries.
• Fast inference: Approximately 4 k documents per second on a V100 GPU (≈ 4100 docs / sec as reported), making it suitable for real‑time re‑ranking pipelines.

Architecture Highlights

• Base model: cross‑encoder/ms‑marco‑MiniLM‑L12‑v2 (quantized variant available).
• Transformer encoder: 12‑layer MiniLM with hidden size 384, 12 attention heads, and a total of ~22 M parameters.
• Classification head: Single linear layer on top of the [CLS] token output, producing a scalar relevance logit.
• Training objective: Pairwise cross‑entropy loss on query‑passage pairs from the MS MARCO passage ranking dataset.

Intended Use Cases

• Re‑ranking the top‑k results from a first‑stage retriever (BM25, DPR, or dense vector search).
• Information‑retrieval pipelines where ranking quality outweighs the need for massive throughput.
• Question‑answering systems that need to select the most relevant passage before answer extraction.
• Search‑as‑you‑type or conversational search where a small candidate set can be evaluated in real time.

Benchmark Performance

The most relevant benchmarks for a cross‑encoder ranker are TREC Deep Learning 2019 (NDCG@10) and the MS MARCO passage re‑ranking set (MRR@10). According to the model card, cross-encoder/ms-marco-MiniLM-L2-v2 achieves:

• NDCG@10 (TREC DL 19): 71.01 %
• MRR@10 (MS MARCO Dev): 34.85 %
• Throughput: ~4 100 documents per second on a V100 GPU.

These numbers place the MiniLM‑L2 variant comfortably between the TinyBERT‑L2 (69.84 % NDCG, 32.56 % MRR) and the larger MiniLM‑L4 (73.04 % NDCG, 37.70 % MRR) models. The modest increase in latency compared to TinyBERT is offset by a noticeable boost in ranking quality, making it a sweet spot for production re‑ranking where both speed and relevance matter.

Hardware Requirements

The MiniLM‑L2 cross‑encoder is designed for efficient inference on modern GPUs while still fitting comfortably on high‑end CPUs.

• VRAM (GPU memory): Roughly 2 GB of GPU memory is sufficient for a batch size of 32–64. Larger batches (e.g., 128) may require up to 4 GB.
• Recommended GPU: NVIDIA V100, A100, RTX 3080/3090, or any GPU with at least 8 GB VRAM for comfortable headroom.
• CPU: A recent multi‑core CPU (e.g., Intel Xeon Gold or AMD EPYC) can run the model via ONNX Runtime or OpenVINO for low‑throughput scenarios; expect ~200–300 ms per 100‑pair batch on a 12‑core CPU.
• Storage: The model files (weights, tokenizer, config) occupy ~200 MB when stored as safetensors or PyTorch binaries. Including the ONNX and OpenVINO variants adds another ~100 MB.
• Performance Characteristics: Inference scales linearly with batch size up to the VRAM limit. For real‑time re‑ranking of ≤ 100 candidates per query, a single V100 can handle > 5 k queries per second.

Use Cases

The ms-marco-MiniLM-L2-v2 cross‑encoder shines in any scenario where a short list of candidate passages must be ordered by relevance.

• Search Engine Re‑ranking: After an initial BM25 or dense vector retrieval, feed the top‑k passages to the model to produce a final ranking that improves click‑through rates.
• Question‑Answering Pipelines: Select the most relevant passage before running a reader model (e.g., BERT‑QA) to extract an answer.
• Enterprise Knowledge Bases: Rank internal documents, policy manuals, or support tickets based on employee queries.
• E‑Commerce Product Search: Re‑rank product descriptions or reviews to surface the most pertinent items for a shopper’s query.
• Legal & Regulatory Research: Prioritise case law passages or regulatory clauses that best match a legal question.

Training Details

The model was trained using the SBERT.net cross‑encoder training script. The pipeline follows a standard pairwise learning‑to‑rank approach:

• Dataset: sentence‑transformers/msmarco – millions of query‑passage pairs with relevance judgments (0 or 1).
• Loss Function: Cross‑entropy loss applied to the binary relevance labels.
• Optimizer: AdamW with a learning rate of 2e‑5 and linear warm‑up over the first 10 % of steps.
• Training Compute: Trained on a single V100 GPU for ~12 hours (≈ 200 k steps, batch size 32).
• Fine‑tuning: Users can further fine‑tune on domain‑specific relevance data by loading the model via CrossEncoder and calling .fit() with a custom dataset.

Related Papers

The model builds on two primary research streams:

• “MiniLM: Deep Self‑Distillation for Task‑Agnostic Compression of Pre‑Trained Transformers” – introduces the MiniLM architecture that underlies this cross‑encoder.
• “MS MARCO: A Large‑Scale Real‑World Dataset for Passage Ranking” – provides the benchmark dataset used for training and evaluation.

These works illustrate the combination of a compact transformer (MiniLM) with a large‑scale relevance‑labelled dataset (MS MARCO), enabling high‑quality ranking while keeping compute requirements modest.

Licensing Information

The model is released under the Apache 2.0 license. This permissive license grants you the freedom to:

• Use the model for commercial or non‑commercial purposes without royalty payments.
• Modify, redistribute, and embed the model in software products.
• Combine the model with other code under different licenses (including proprietary code).

Key requirements are:

• Preserve the original copyright notice and license text in any redistribution.
• Provide a clear attribution to the original authors (cross‑encoder) and the MS MARCO dataset.
• State any modifications you make to the model or its weights.

Because the license is permissive, there are no “unknown” restrictions—developers can safely integrate the model into SaaS platforms, on‑premise search engines, or mobile applications, provided they comply with the attribution clause.