roberta-base-squad2

Technical Overview

What is this model? deepset/roberta-base-squad2 is an English‑language, extractive question‑answering (QA) model built on top of the roberta‑base transformer. It has been fine‑tuned on the SQuAD 2.0 dataset, which contains both answerable and unanswerable questions. The model predicts the start and end token positions of the answer span within a provided context, returning “no answer” when the question is unanswerable.

Key features and capabilities

• Exact‑match (EM) ≈ 79.93 % and F1 ≈ 82.95 % on the SQuAD 2.0 validation set.
• Supports the question‑answering pipeline in Hugging Face Transformers and in the Haystack framework.
• Works out‑of‑the‑box for English documents up to 386 tokens (including the query).
• Provides a “no‑answer” prediction for unanswerable queries, a core requirement for real‑world QA systems.
• Distributed under a Creative Commons Attribution‑4.0 (CC‑BY‑4.0) license, enabling open reuse with attribution.

Architecture highlights

• Base model: FacebookAI/roberta‑base, a 12‑layer, 125 M‑parameter transformer pre‑trained on 160 GB of English text.
• Fine‑tuning head: a linear layer that maps the hidden state of each token to start‑ and end‑logits.
• Training hyper‑parameters: batch size = 96, learning rate = 3 × 10⁻⁵, linear warm‑up (20 % of steps), 2 epochs, max sequence length = 386, doc‑stride = 128, max query length = 64.
• Optimized for both PyTorch and TensorFlow (and JAX via the transformers library), with safetensors weight format for faster loading.

Intended use cases

• Customer‑support bots that need to retrieve precise answers from knowledge‑base articles.
• Search‑engine snippets that highlight the exact answer within a document.
• Academic research tools that extract evidence passages from scientific papers.
• Any downstream application that requires fast, on‑device extractive QA without a large language model.

Benchmark Performance

For extractive QA, the most informative metrics are Exact Match (EM) and F1 score. EM measures the percentage of predictions that exactly match the ground‑truth answer span, while F1 balances precision and recall at the token level.

The deepset/roberta-base-squad2 model achieves:

• SQuAD 2.0 (validation) – EM = 79.93 %, F1 = 82.95 % (verified tokens provided).
• SQuAD 1.1 (validation) – EM = 85.29 %, F1 = 91.84 %.
• Adversarial QA (validation) – EM = 29.5 %, F1 = 40.37 %.
• Adversarial SQuAD (validation) – EM = 78.57 %, F1 = 84.47 %.
• SQuAD‑Shifts (test) – EM ranging from 69.92 % (Amazon) to 82.93 % (NYT), F1 ranging from 82.94 % to 90.76 %.

These results demonstrate that the model retains strong performance on the original SQuAD benchmark while also handling unanswerable questions and domain shifts reasonably well. Compared to the base roberta‑base model (which is not fine‑tuned for QA), the fine‑tuned version shows a > 30 % absolute gain in EM and F1 on SQuAD 2.0, confirming the value of task‑specific adaptation.

Hardware Requirements

VRAM for inference

• Model size: ~ 500 MB (safetensors format) – fits comfortably on a single GPU with ≥ 8 GB VRAM.
• Typical batch size = 1 (real‑time QA) uses ~ 2 GB VRAM; batch size = 16 (throughput‑oriented) stays under 6 GB.

Recommended GPU

• Any modern NVIDIA GPU with at least 8 GB memory (e.g., RTX 3060, Tesla V100, A100).
• For high‑throughput production, a V100‑class GPU (16‑32 GB) or A100 (40 GB) enables larger batch sizes and lower latency.

CPU & Storage

• CPU: 4‑core modern processor is sufficient for preprocessing; inference can be delegated to the GPU.
• Disk: ~ 600 MB for model weights, plus additional space for tokenizers and optional quantized versions.
• RAM: 8 GB+ to hold the tokenizer and intermediate tensors.

Performance characteristics

• Latency: ~ 30‑45 ms per query on a V100 for a 386‑token context.
• Throughput: > 200 queries / second on a single A100 when batching 32‑64 inputs.

Use Cases

Primary applications

• Customer‑service chatbots that need to pull exact answers from product manuals or FAQ pages.
• Enterprise knowledge‑base search where a user asks a natural‑language question and receives a highlighted answer span.
• Legal or compliance tools that extract specific clauses from contracts.
• Educational platforms that provide instant answers from textbook passages.

Real‑world examples

• Tech support: A user asks “How do I reset my router?” and the system returns the exact sentence from the troubleshooting guide.
• Healthcare FAQ: Patients query “What are the side effects of drug X?” and receive a concise answer from the drug’s label.
• Financial documents: Analysts extract “net profit” figures from quarterly reports without manual scanning.

Integration possibilities

• Directly via the Hugging Face pipeline API.
• Within the Haystack framework for scalable, multi‑document retrieval‑augmented QA pipelines.
• Wrapped in a REST endpoint using FastAPI or Flask for micro‑service deployment.
• Exported to ONNX or TorchScript for edge‑device inference.

Training Details

Methodology

• Fine‑tuning was performed on the SQuAD 2.0 dataset, which contains 20 k training examples (including unanswerable questions).
• Training ran for 2 epochs over the full dataset with a batch size of 96, using a linear learning‑rate schedule with a 20 % warm‑up period.
• Sequence length was capped at 386 tokens (64 for the query, 322 for the context) and a document stride of 128 tokens ensured overlapping windows for long passages.
• Optimisation used AdamW with a learning rate of 3 × 10⁻⁵.

Compute resources

• Four NVIDIA Tesla V100 GPUs (each 16 GB) were used, providing sufficient parallelism for the large batch size.
• Training time was roughly 3‑4 hours per epoch, resulting in a total wall‑clock time of ~ 7‑8 hours.

Fine‑tuning capabilities

• Because the model is a standard transformers checkpoint, you can further fine‑tune it on domain‑specific QA data (e.g., medical or legal corpora) using the same hyper‑parameters.
• Supports mixed‑precision (FP16) training for faster convergence on modern GPUs.

Related Papers

The model builds on two foundational works:

• RoBERTa: A Robustly Optimized BERT Pretraining Approach – the underlying architecture (Facebook AI, 2019).
• SQuAD 2.0: A Dataset for Machine Comprehension with Unanswerable Questions – the fine‑tuning dataset (Rajpurkar et al., 2019).

These papers introduce the transformer improvements that make RoBERTa more data‑efficient and the evaluation methodology that forces models to recognize when a question has no answer, a capability that deepset/roberta-base-squad2 inherits.

Licensing Information

The model card lists the license as cc-by-4.0 (Creative Commons Attribution 4.0) but also marks the overall license as “unknown”. In practice, the CC‑BY‑4.0 terms apply to the model weights and the associated code.

What this means for users

• Commercial use: Allowed, provided you give appropriate credit to the original authors (deepset) and to the underlying roberta‑base model.
• Modification: You may adapt, fine‑tune, or redistribute the model, again with attribution.
• Restrictions: No additional restrictions beyond the standard CC‑BY‑4.0 clause (e.g., you cannot impose further legal constraints on downstream users).

Attribution requirement

When using the model in a product, paper, or public demo, include a citation such as:

deepset. (2023). roberta-base-squad2. Hugging Face. https://huggingface.co/deepset/roberta-base-squad2

If you distribute the model (e.g., on a hardware device), the same attribution must be retained in the accompanying documentation.