bert-mini-finetuned-mnli

Technical Overview

The bert-mini-finetuned-mnli model is a compact BERT‑mini transformer (≈ 4 M parameters) that has been fine‑tuned on the Multi‑Genre Natural Language Inference (MNLI) dataset using the novel second‑order optimizer M‑FAC. Its primary purpose is text‑classification – specifically, predicting whether a premise entails, contradicts, or is neutral to a hypothesis.

Key features and capabilities:

• Lightweight architecture: BERT‑mini retains the transformer encoder stack but reduces hidden size and depth, enabling fast inference on modest hardware.
• Second‑order optimization: M‑FAC approximates curvature information without storing large matrices, leading to higher accuracy than a vanilla Adam baseline on the same hyper‑parameter budget.
• MNLI‑ready: The model achieves 75.13 % matched and 75.93 % mismatched accuracy on the MNLI validation set – a noticeable gain over the Adam‑tuned counterpart.
• PyTorch & Hugging Face Transformers compatible: Ready to drop into the standard pipeline('text‑classification') API.
• Azure‑deployable: Tagged for Azure endpoints, making it easy to host as a scalable REST service.

Architecture highlights:

• 4 encoder layers, 256 hidden dimension, 4 attention heads.
• WordPiece tokenizer with a 30 k vocabulary (inherited from the original BERT‑mini).
• Maximum sequence length of 128 tokens (as used during fine‑tuning).
• Fine‑tuned with a learning rate of 1e‑4, 1024 accumulated gradients, and a dampening factor of 1e‑6 under the M‑FAC optimizer.

Intended use cases include any downstream task that benefits from fast, on‑device inference of natural‑language inference, such as real‑time content moderation, chatbot response validation, or lightweight QA pipelines.

Benchmark Performance

For a natural‑language inference model, the MNLI benchmark is the de‑facto standard because it tests the ability to reason across multiple genres and both matched/mismatched domains. The README reports the following validation scores for the best of five runs:

• Matched accuracy: 75.13 %
• Mismatched accuracy: 75.93 %

A direct comparison with the same architecture trained with Adam shows a clear advantage:

These results matter because they demonstrate that a second‑order optimizer can extract more performance from a tiny transformer without increasing model size or training time. Compared with other “mini” BERT variants (e.g., BERT‑tiny), the M‑FAC‑fine‑tuned BERT‑mini sits at the top of the accuracy‑vs‑size trade‑off curve, making it attractive for latency‑critical applications.

Hardware Requirements

• VRAM for inference: ~1 GB of GPU memory is sufficient for a single‑batch inference at sequence length 128. The model can also run on CPUs with < 2 GB RAM, though latency will increase.
• Recommended GPU: Any modern NVIDIA GPU with ≥ 4 GB VRAM (e.g., RTX 2060, GTX 1660 Super, or Tesla T4) provides sub‑10 ms latency per request when using the Hugging Face pipeline API.
• CPU: 8‑core Intel i5/AMD Ryzen 5 or higher; SIMD‑enabled PyTorch builds will keep inference under 50 ms per sentence.
• Storage: Model files (config, tokenizer, weights) total ≈ 150 MB. Including the Hugging Face cache and optional example scripts, allocate ~300 MB of disk space.
• Performance characteristics: With batch size = 32 on a RTX 2070, throughput reaches ~250 tokens / second, which is ample for most real‑time services.

Use Cases

Because the model excels at natural‑language inference while remaining lightweight, it fits naturally into several real‑world scenarios:

• Content moderation: Quickly determine if a user‑generated statement contradicts platform policies.
• Chatbot response validation: Verify that a bot’s reply logically follows the user’s query before sending it.
• Document similarity & retrieval: Use the entailment scores to rank passages that support or refute a claim.
• Edge‑device NLP: Deploy on smartphones, IoT gateways, or embedded servers where memory and power are limited.
• Azure AI endpoints: The model is tagged for Azure deployment, enabling scalable, managed inference via Azure Machine Learning or Azure Functions.

Training Details

The fine‑tuning pipeline mirrors the official Hugging Face GLUE text‑classification example. The only deviation is the replacement of the default Adam optimizer with the M‑FAC optimizer. Hyper‑parameters used for all runs (including BERT‑tiny and BERT‑mini) were:

• Learning rate: 1e‑4
• Number of accumulated gradients: 1024
• Dampening factor: 1e‑6
• Batch size per device: 32
• Maximum sequence length: 128
• Training epochs: 5

The training script (run on a single CUDA device) can be invoked with the following command:

CUDA_VISIBLE_DEVICES=0 python run_glue.py \
  --seed 8276 \
  --model_name_or_path prajjwal1/bert-mini \
  --task_name mnli \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 1e-4 \
  --num_train_epochs 5 \
  --output_dir out_dir/ \
  --optim MFAC \
  --optim_args '{"lr": 1e-4, "num_grads": 1024, "damp": 1e-6}'

The authors note that modest hyper‑parameter tuning (e.g., batch size, learning rate, number of gradients) could push the validation scores even higher. The model can be further fine‑tuned on downstream tasks (e.g., SQuAD v2, other GLUE tasks) using the same M‑FAC optimizer.

Related Papers

The core innovation behind this model is the M‑FAC: Efficient Matrix‑Free Approximations of Second‑Order Information paper (NeurIPS 2021) by Frantar, Kurtic, and Alistarh. M‑FAC provides a curvature‑aware optimization step that improves convergence for small transformers without the memory overhead of full Fisher or Hessian matrices. The MNLI dataset itself is described in the original GLUE benchmark paper, which serves as the evaluation foundation for this fine‑tuning effort.

Licensing Information

The model card lists the license as unknown. In practice, an “unknown” license means the repository does not explicitly state a permissive or restrictive term. Consequently:

• Commercial use: You should treat the model as “all‑rights‑reserved” until you obtain explicit permission from the author (M‑FAC) or verify the license in the underlying M‑FAC source repository.
• Restrictions: Without a clear license, redistribution, modification, or incorporation into proprietary products may be prohibited under default copyright law.
• Attribution: Even when the license is unclear, best practice is to credit the original creators. Include a citation to the M‑FAC paper (see “Related Papers”) and a link to the Hugging Face model card.
• Due diligence: Before deploying in a production environment, contact the model owner or check the GitHub repository for a LICENSE file. If a permissive license (e.g., MIT, Apache 2.0) is later added, you can safely use the model commercially.