deberta-xlarge-mnli

Technical Overview

Model ID: microsoft/deberta-xlarge-mnli
Model Name: DeBERTa‑xlarge‑MNLI
Author: Microsoft
Pipeline Tag: text‑classification

DeBERTa‑xlarge‑MNLI is a 750 million‑parameter, transformer‑based language model that builds on the DeBERTa‑v1 architecture and has been fine‑tuned on the Multi‑Genre Natural Language Inference (MNLI) dataset. Its primary function is to predict the relationship between a premise and a hypothesis (Entailment, Contradiction, or Neutral) and, because MNLI is a strong “universal” NLU task, the model can be re‑used for a wide range of downstream text‑classification problems via simple fine‑tuning.

• Key Features:
  • Disentangled attention that separates content and position information.
  • Enhanced mask decoder that improves masked‑language‑model pre‑training.
  • Large‑scale pre‑training on ~80 GB of text data.
  • Fine‑tuned on MNLI, giving strong zero‑shot transfer to other NLU benchmarks.
• Architecture Highlights:
  • 12 transformer layers with a hidden size of 1 024 and 16 attention heads.
  • Relative positional encodings and disentangled attention (content‑based and position‑based queries).
  • Layer‑norm and GELU activation, following the BERT‑style design but with the DeBERTa‑specific modifications.
• Intended Use Cases:
  • General‑purpose text classification (sentiment analysis, topic detection, intent classification).
  • Natural language inference and sentence‑pair tasks.
  • Feature extraction for downstream QA, summarisation, or retrieval pipelines.

Benchmark Performance

The model’s performance is reported on the GLUE benchmark and several QA datasets. The most relevant benchmark for this model is MNLI, where it achieves 91.5 % accuracy (matched) and 91.2 % accuracy (mismatched). In the GLUE table from the README, DeBERTa‑xlarge‑MNLI also shows competitive scores on SST‑2 (97.0 % accuracy) and RTE (93.1 % accuracy). Compared to BERT‑Large (86.6 % MNLI) and RoBERTa‑Large (90.2 % MNLI), the DeBERTa‑xlarge‑MNLI model delivers a clear improvement while staying within a similar parameter budget.

These benchmarks matter because MNLI is a strong proxy for general language understanding; high MNLI scores usually translate to better transfer performance on other downstream tasks such as sentiment analysis, paraphrase detection, and QA. The model’s scores on SQuAD 1.1/2.0 (not directly reported for the xlarge variant) are expected to be on par with the DeBERTa‑Large baseline, which already outperforms BERT and RoBERTa on those tasks.

Hardware Requirements

DeBERTa‑xlarge‑MNLI is a 750 M‑parameter model, so inference and fine‑tuning demand substantial GPU memory. Typical VRAM requirements are:

• Inference (batch‑size = 1): ~12 GB GPU memory.
• Fine‑tuning (batch‑size = 8–16): 24 GB – 32 GB GPU memory (e.g., NVIDIA RTX 3090, A100 40 GB, or V100 32 GB).

For CPU‑only inference, expect latency in the range of 1–2 seconds per sentence on a modern 8‑core Xeon processor, but the model will be memory‑bound and may require > 30 GB RAM to hold the weights. Storage footprint is roughly 3 GB for the model checkpoint (including tokenizer files). Using mixed‑precision (FP16) can halve the memory consumption on supported GPUs.

Use Cases

DeBERTa‑xlarge‑MNLI shines in any scenario that needs high‑accuracy sentence‑pair classification or strong general‑purpose text classification. Typical applications include:

• Customer support automation: Detecting intent and sentiment from user queries.
• Legal and compliance review: Identifying contradictory statements across contracts.
• Content moderation: Classifying user‑generated content for policy violations.
• Knowledge‑base retrieval: Ranking candidate answers based on premise‑hypothesis similarity.

The model can be integrated via the Hugging Face transformers library, deployed on Azure Machine Learning, or wrapped in a REST API for low‑latency inference.

Training Details

The base DeBERTa‑xlarge model was pre‑trained on ~80 GB of English text using a masked language modeling objective with the disentangled attention scheme. After pre‑training, the model was fine‑tuned on the MNLI dataset (≈ 393 k sentence pairs) for three epochs with a learning rate of 2e‑5 and a batch size of 32, following the standard GLUE fine‑tuning recipe. The fine‑tuned checkpoint is the one hosted at https://huggingface.co/microsoft/deberta-xlarge-mnli.

Key training resources:

• GPU: 8 × NVIDIA V100 32 GB (or equivalent) for the pre‑training phase.
• Training time: ~ 2 weeks of mixed‑precision training for the base model; MNLI fine‑tuning takes < 1 hour on a single A100.
• Framework: PyTorch with the Hugging Face transformers library.

The model is fully compatible with the Hugging Face pipeline('text-classification') API and can be further fine‑tuned on any downstream classification task (e.g., SST‑2, QNLI, RTE) using the same hyper‑parameter settings.

Related Papers

• DeBERTa: Decoding‑enhanced BERT with Disentangled Attention – He et al., 2021 (arXiv:2006.03654).
• DEBERTA: DECODING‑ENHANCED BERT WITH DISENTANGLED ATTENTION – Conference on Learning Representations (ICLR) 2021.

These papers introduce the disentangled attention mechanism and the enhanced mask decoder that give DeBERTa its performance edge over BERT and RoBERTa.

Licensing Information

The README lists the license as MIT, but the model card metadata shows license: unknown. In practice, Microsoft has released the DeBERTa‑v1 family under the MIT license, which permits:

• Free commercial and non‑commercial use.
• Modification, distribution, and private use.
• No warranty or liability.

If the license is truly “unknown”, you should treat the model as read‑only for research and internal prototyping until you can confirm the MIT terms from the official repository. Attribution to the original paper (He et al., 2021) is required, and any redistribution must retain the original copyright notice. No additional restrictions (e.g., “no‑commercial”) are indicated in the README.