Technical Overview
The facebook/dpr-ctx_encoder-single-nq-base model is the Context Encoder component of Facebook’s Dense Passage Retrieval (DPR) framework. DPR is a two‑tower architecture designed for open‑domain question answering (QA) where a question encoder maps a query into a dense vector and a context encoder maps each candidate passage (e.g., a Wikipedia paragraph) into the same vector space. Retrieval is performed by nearest‑neighbor search, returning the most relevant passages for a downstream reader model.
Key capabilities of the ctx_encoder‑single‑nq‑base include:
- Dense passage embeddings – Produces 768‑dimensional vectors that capture semantic similarity beyond keyword matching.
- Fast, low‑latency retrieval – Allows indexing of millions of passages and sub‑second lookup using FAISS or ScaNN.
- Compatibility with DPR pipeline – Works out‑of‑the‑box with the matching question encoder (
dpr‑question‑encoder‑single‑nq‑base) and reader (dpr‑reader‑single‑nq‑base). - English‑only, BERT‑base architecture – 12 transformer layers, 110 M parameters, pre‑trained on the Natural Questions (NQ) dataset.
Architecture highlights:
- Based on the BERT‑base uncased transformer.
- Uses the
pooler_output(the CLS token after a dense layer) as the passage embedding. - Trained with a contrastive loss that pushes positive passage–question pairs together while pulling negative passages apart.
- Single‑NQ variant: trained exclusively on the Natural Questions open‑domain split, making it especially suited for web‑scale QA.
Intended use cases revolve around open‑domain QA pipelines, semantic search, and any application that requires fast, high‑quality passage retrieval from a large corpus. The model is also useful for knowledge‑grounded dialogue and retrieval‑augmented generation (RAG) where a dense retriever is a core component.
Benchmark Performance
Because DPR is a retrieval‑oriented model, the most relevant benchmarks evaluate recall at K (R@K) on large question‑answer datasets. The original DPR paper reports the following numbers on the Natural Questions (NQ) test set using a 100‑passage retrieval budget:
- R@10: ~78 %
- R@20: ~84 %
- R@100: ~93 %
These metrics matter because they directly affect the downstream reader’s ability to locate the correct answer span. Higher recall means the reader sees more relevant passages, leading to higher exact‑match (EM) and F1 scores for the full QA system.
When compared to earlier dense retrievers (e.g., ColBERT) and classic BM25 baselines, DPR’s single‑NQ encoder consistently outperforms sparse methods on semantic similarity while requiring comparable inference latency when paired with an efficient ANN index.
Hardware Requirements
The ctx_encoder‑single‑nq‑base model contains ~110 M parameters (≈420 MB on disk). For inference, the following hardware guidelines apply:
- GPU VRAM: 4 GB is sufficient for a single forward pass with batch size = 1 (the model loads ~2 GB of GPU memory). Batch sizes of 8–16 comfortably fit on 8 GB‑VRAM GPUs (e.g., RTX 3060, RTX 2070).
- Recommended GPUs: NVIDIA RTX 3070/3080, A100, or any GPU with ≥6 GB VRAM for high‑throughput batch inference.
- CPU inference: Possible on modern CPUs (e.g., Intel i7‑12700K) but will be 3‑5× slower than GPU. A minimum of 16 GB RAM is advised for tokenization and model loading.
- Storage: Model files (~420 MB) plus tokenizer vocab (~120 MB) – total < 600 MB. Disk space for the passage index (FAISS) depends on corpus size (e.g., 100 M passages ≈ 30 GB).
- Performance: On an RTX 3080, a single query with a batch of 32 passages takes ~2 ms for embedding generation; indexing 1 M passages takes ~30 seconds per pass.
Use Cases
The DPR context encoder shines in scenarios where fast, high‑quality passage retrieval is a bottleneck. Typical applications include:
- Open‑Domain Question Answering: Powering the retrieval stage of systems like DPR or Retrieval‑Augmented Generation (RAG).
- Semantic Search Engines: Enabling search over large knowledge bases (Wikipedia, corporate documents) with meaning‑based matching rather than keyword matching.
- Chatbots & Conversational AI: Providing context‑aware answers by retrieving relevant passages before generation.
- Legal & Medical Document Retrieval: Finding statutes, case law, or clinical guidelines that match a user query semantically.
- Recommendation Systems: Matching user queries to product descriptions or help‑center articles.
Integration is straightforward with the transformers library: load the tokenizer and model, encode passages offline, store embeddings in an ANN index, and query at runtime. The model also works seamlessly with Hugging Face’s DPR pipeline utilities.
Training Details
The ctx_encoder‑single‑nq‑base was trained on the Natural Questions (NQ) open‑domain split. The dataset consists of real Google search queries paired with Wikipedia passages that contain the answer span. Training steps:
- Data preparation: Positive passage–question pairs were sampled from the NQ annotations; hard negatives were mined from the top‑k BM25 results that did not contain the answer.
- Model initialization: Weights were initialized from the BERT‑base uncased checkpoint.
- Loss function: A contrastive (in‑batch) loss that maximizes the dot‑product similarity of positive pairs while minimizing it for negatives.
- Optimization: AdamW optimizer with a learning rate of 3e‑5, linear warm‑up for 10 % of steps, and weight decay of 0.01.
- Compute: Trained on 8 × NVIDIA V100 GPUs for ~3 days (≈ 200 k steps, batch size = 64 per GPU). Total compute ≈ 1.5 M GPU‑hours.
- Fine‑tuning: The model can be further fine‑tuned on domain‑specific corpora (e.g., legal or biomedical texts) by continuing the contrastive training with domain‑specific positive/negative passages.
Licensing Information
The model is released under the Creative Commons Attribution‑NonCommercial 4.0 International (CC‑BY‑NC‑4.0) license. This license permits:
- Free non‑commercial use, sharing, and adaptation of the model weights and tokenizer.
- Requirement to retain attribution to the original creators (Facebook AI) and to cite the associated research papers.
Commercial use is not allowed without obtaining a separate commercial license from the rights holder. Users must also include a copy of the CC‑BY‑NC‑4.0 license text in any distribution of the model or derived works.
If you plan to integrate the encoder into a product that generates revenue (e.g., SaaS, enterprise search), you should contact Facebook AI for a commercial licensing agreement. For academic, research, or personal projects the current license is fully permissive as long as the non‑commercial clause is respected.