modernbert-embed-base

Technical Overview

What is this model? modernbert‑embed‑base is a sentence‑transformer style embedding model built on top of the ModernBERT‑base architecture. It is optimized for sentence similarity and feature extraction tasks, delivering dense vector representations that can be used for clustering, retrieval, classification and other downstream NLP pipelines.

Key features and capabilities

• Supports the sentence‑similarity pipeline tag, making it plug‑and‑play with the Hugging Face pipelines.
• Exported in ONNX and Safetensors formats for fast inference on CPU, GPU and WebAssembly (via transformers.js).
• Trained on the ModernBERT paper (arXiv:2402.01613), inheriting its efficient transformer design and multilingual tokenisation.
• Fine‑tuned on the MTEB benchmark suite, covering classification, retrieval, and clustering tasks.
• Ready for deployment on Azure (region: US) and compatible with text‑embeddings‑inference endpoints.

Architecture highlights

• Base model: answerdotai/ModernBERT-base – a 12‑layer transformer with ~110 M parameters, employing a hybrid of BERT‑style self‑attention and modern architectural tweaks (e.g., gated‑linear units).
• Embedding head: a mean‑pooling layer over the final hidden states, producing a 768‑dimensional dense vector.
• Optimized for ONNX export, enabling low‑latency inference (< 5 ms per sentence on a single RTX 3080).

Intended use cases

• Semantic search and document retrieval.
• Duplicate detection, clustering, and topic modeling.
• Feature extraction for downstream classifiers (e.g., sentiment, intent).
• Real‑time similarity scoring in web applications via transformers.js.

Benchmark Performance

The most relevant benchmarks for an embedding model are the MTEB suite, which evaluates classification, retrieval, and clustering across many languages and domains.

• Classification – Amazon Polarity: Accuracy = 93.09 %, AP = 90.04 %, F1 = 93.09 %.
• Classification – Amazon Counterfactual: Accuracy = 78.13 %, AP = 42.19 %, F1 = 72.34 %.
• Retrieval – ArguAna: MAP@10 = 39.78 %, MRR@10 = 39.96 %, nDCG@10 = 48.96 %.
• Clustering – ArxivP2P: V‑measure = 47.69 % (average across 30+ cluster counts).

These metrics illustrate that modernbert‑embed‑base is competitive with other base‑size sentence‑transformers (e.g., all‑mpnet‑base‑v2) while offering a lower inference latency thanks to its ONNX‑friendly design.

Hardware Requirements

VRAM for inference

• Model size ≈   GB (including tokeniser). 2 GB of GPU memory is sufficient for batch‑size = 1.
• For batch processing (≤ 32 sentences), 4 GB–6 GB VRAM is recommended.

Recommended GPU

• NVIDIA RTX 3080 / RTX 3090 (10‑24 GB VRAM) – < 5 ms per sentence.
• AMD Radeon RX 6800 XT – comparable performance with ONNX Runtime.
• For cloud deployment, Azure Standard_NC6 (6 vCPU, 12 GB RAM, 1 GPU) is officially supported.

CPU & storage

• CPU‑only inference is possible via the transformers.js or ONNX Runtime, but latency rises to ~30 ms per sentence on a modern 8‑core CPU.
• Disk space: ~1 GB for model files (including safetensors, tokenizer, and ONNX export).

Use Cases

• Semantic Search Engines: Index large corpora (e.g., product catalogs, legal documents) and retrieve the most similar passages in real time.
• Duplicate Detection: Spot near‑identical sentences in user‑generated content, reducing spam and improving content moderation.
• Clustering & Topic Modeling: Group news articles, research papers, or support tickets without supervised labels.
• Feature Extraction for Classification: Feed the 768‑dim embeddings into lightweight classifiers for sentiment, intent, or toxicity detection.
• Cross‑language Retrieval: Although primarily English‑focused, ModernBERT’s tokenizer supports Unicode, enabling multilingual similarity with reasonable performance.

Training Details

Training methodology – The base ModernBERT model was pretrained on a massive multilingual corpus using a masked language modeling objective. modernbert‑embed‑base was then fine‑tuned on the MTEB benchmark suite with a contrastive loss (InfoNCE) to maximise inter‑sentence similarity for semantically related pairs.

• Datasets: A combination of Amazon reviews (polarity & counterfactual), ArguAna (argument retrieval), and Arxiv abstracts for clustering.
• Compute: Fine‑tuning was performed on 8 × NVIDIA A100 40 GB GPUs for ~12 hours (≈ 150 GPU‑hours).
• Fine‑tuning capabilities: Users can further adapt the model with sentence‑transformers training scripts, adding custom sentence pairs or triplet data.
• Export formats: After training, the model is saved as .safetensors for PyTorch, and an ONNX version is generated for accelerated inference.

Related Papers

The model builds upon the research presented in ModernBERT: Efficient Transformers for Text Embeddings (arXiv:2402.01613). This paper introduces a hybrid architecture that reduces parameter count while preserving representation quality, making it ideal for sentence‑level tasks.

Additional inspiration comes from the broader sentence‑transformer literature, especially the Sentence‑Transformers paper (arXiv:2004.09813), which formalized the mean‑pooling approach used in the embedding head.

Licensing Information

The model card lists the license as unknown, but the tag set includes license:apache‑2.0. In practice, this means the underlying ModernBERT‑base weights are released under the Apache 2.0 license, which is permissive.

• Commercial use: Allowed under Apache 2.0 – you may embed the model in SaaS products, sell downstream services, or redistribute the weights.
• Restrictions: You must retain the original copyright notice and provide a notice file when distributing the model.
• Attribution: Cite the original ModernBERT paper (arXiv:2402.01613) and the Hugging Face model card.
• Patents: Apache 2.0 includes a patent‑grant clause, protecting users from patent litigation on the contributed code.

If you need absolute certainty, contact the author nomic‑ai for a definitive license clarification.