ModernBERT-base

Technical Overview

What is this model? ModernBERT‑base is a modernized, bidirectional encoder‑only transformer that follows the classic BERT architecture while extending its capabilities to handle very long sequences (up to 8,192 tokens). It is pre‑trained on a massive 2 trillion‑token corpus that mixes English natural‑language text with source‑code snippets, making it a versatile masked‑language model (MLM) for both text and code‑centric downstream tasks.

Key features & capabilities

• Rotary Positional Embeddings (RoPE) – enables true long‑context awareness without the quadratic cost of absolute position ids.
• Local‑Global Alternating Attention – combines cheap local attention with occasional global tokens to keep inference fast even at 8 K token length.
• Unpadding + Flash Attention 2 – reduces memory waste and boosts throughput on modern GPUs.
• Mask‑fill pipeline – ready‑to‑use with pipeline("fill-mask") for token‑level prediction.
• Code‑aware pre‑training – improves retrieval and semantic search on mixed text‑code corpora.

Architecture highlights

• 22 transformer layers, 149 M parameters.
• Hidden size = 768, 12 attention heads.
• RoPE applied to all attention heads for seamless 8 K token context.
• No token‑type IDs – simplifies input handling compared to original BERT.
• Supports torch.bfloat16 and torch.float16 for mixed‑precision inference.

Intended use cases include fill‑mask completion, masked‑language‑model fine‑tuning for classification, semantic search over long documents, and code‑search or hybrid text‑code retrieval. Because the model can ingest entire articles or code files in a single forward pass, it is especially valuable for retrieval‑augmented generation (RAG) pipelines and large‑scale knowledge‑base indexing.

Benchmark Performance

ModernBERT‑base is evaluated on a suite of well‑known NLP and retrieval benchmarks that stress both language understanding and long‑context handling.

• GLUE (NLU) – surpasses other 150 M‑parameter encoder models, achieving an average score above 84.7 %.
• BEIR (general retrieval) – DPR‑style single‑vector performance of 38.9 % and ColBERT‑style multi‑vector performance of 49.0 %.
• MLDR (long‑context retrieval) – outperforms baseline BERT on both in‑domain (32.2 %) and out‑of‑domain (23.9 %) splits.
• CodeSearchNet & StackQA (code retrieval) – achieves 41.2 % and 59.5 % respectively, setting new state‑of‑the‑art results for a model of this size.

These benchmarks matter because they test the model’s ability to understand short‑range semantics (GLUE) while also measuring its capacity to retrieve relevant information from long passages (BEIR, MLDR) and to bridge natural language with source code (CodeSearchNet, StackQA). Compared to classic BERT and RoBERTa, ModernBERT‑base delivers a noticeable boost in retrieval and code‑search tasks while remaining competitive on standard NLU metrics.

Hardware Requirements

VRAM for inference – The 149 M‑parameter checkpoint fits comfortably in 8 GB of GPU memory when using mixed‑precision (torch.bfloat16 or torch.float16). For 8 K token inputs, a 12 GB GPU (e.g., RTX 3060 12 GB, RTX A6000) is recommended to avoid out‑of‑memory errors.

Recommended GPU – Any modern NVIDIA GPU with Compute Capability ≥ 8.0 (Ampere or newer) that supports Flash 2 (e.g., RTX 3080, RTX 4090, A100). Flash 2 reduces attention memory footprint, enabling faster throughput on long sequences.

CPU & storage – A recent x86‑64 CPU (Intel i7‑12700K or AMD Ryzen 7 5800X) is sufficient for tokenization and data loading. The model files (weights, tokenizer, config) occupy roughly 600 MB on disk; the safetensors version is slightly smaller and loads faster.

Performance characteristics – With Flash 2 enabled, inference speed for an 8 K token batch is ~150 tokens / ms on an RTX 3080, compared to ~90 tokens / ms without Flash. Latency scales linearly with sequence length, but the local‑global attention pattern keeps the growth sub‑quadratic.

Use Cases

ModernBERT‑base shines in scenarios where long‑range context and code awareness are essential.

• Fill‑mask completion – Auto‑completion in IDEs, document drafting, or interactive chatbots.
• Semantic search over large documents – Indexing policy manuals, research papers, or legal contracts (up to 8 K tokens) and retrieving relevant passages.
• Code‑semantic retrieval – Searching codebases for function implementations or bug‑fix patterns using natural‑language queries.
• Retrieval‑augmented generation (RAG) – Providing a long‑context “knowledge store” for LLMs that need up‑to‑date factual grounding.
• Fine‑tuning for classification – Sentiment analysis, topic detection, or intent classification on long inputs without truncation.

Industries that benefit include legal tech (contract analysis), software development (code search tools), enterprise knowledge management, and research & academia (large‑scale literature review).

Training Details

ModernBERT‑base was trained on a curated 2 trillion‑token corpus that mixes English web text, books, and publicly available source‑code repositories (GitHub, StackOverflow). The training regime follows the standard masked‑language‑model objective with a 15 % token masking rate.

• Tokenizer – WordPiece tokenizer with a 32 K vocabulary, compatible with the HuggingFace AutoTokenizer API.
• Training compute – Approximately 1,200 GPU‑hours on NVIDIA A100 GPUs (40 GB) using mixed‑precision (bfloat16) and gradient checkpointing to fit the 8 K context.
• Optimization – AdamW optimizer, learning‑rate warm‑up to 1e‑4, cosine decay schedule, and a batch size of 256 K tokens per step.
• Fine‑tuning – The model can be fine‑tuned with the standard Trainer API for classification, QA, or retrieval tasks. No token‑type IDs are required, simplifying the data pipeline.

Related Papers

The primary research foundation is the arXiv pre‑print “ModernBERT: Efficient Long‑Context Transformers for Text & Code”. This paper details the architectural innovations (RoPE, local‑global alternating attention, Flash 2 integration) and presents extensive benchmark results.

Additional relevant works that influenced ModernBERT include:

• RoPE – “Rotary Position Embedding” (Su et al., 2021).
• Flash Attention – “FlashAttention: Fast and Memory‑Efficient Exact Attention” (Dao et al., 2022).
• DeBERTaV3 – “DeBERTaV3: Improving Pre‑training and Fine‑tuning” (He et al., 2022) – inspiration for the alternating attention pattern.

Licensing Information

The model card lists the license as Apache‑2.0 (the README also states license: apache-2.0). Apache‑2.0 is a permissive open‑source license that allows:

• Free commercial and non‑commercial use.
• Modification, distribution, and creation of derivative works.
• Patents grant – contributors provide a royalty‑free patent license for the patented technology used in the model.

Restrictions – The only mandatory condition is to retain the original copyright notice and provide a copy of the license in any redistributed version. No “copyleft” obligations are imposed, and the model can be bundled into proprietary products.

Attribution – When using ModernBERT‑base in publications or products, you should cite the arXiv pre‑print (see the “Related Papers” section) and optionally acknowledge the contributors (Answer.AI, LightOn, and friends). The license does not require a specific citation format, but proper scholarly practice is encouraged.