distilbert-base-uncased-emotion

Technical Overview

What is this model? distilbert‑base‑uncased‑emotion is a compact, English‑language transformer that has been fine‑tuned for emotion detection. It takes a raw text string as input and returns a probability distribution over six basic emotions (anger, fear, joy, love, sadness, surprise) as defined in the Emotion dataset.

Key features & capabilities

• Built on DistilBERT – a 40 % smaller, 60 % faster variant of BERT that retains 97 % of its language understanding.
• 66 M parameters, 768‑dimensional hidden size, 12 transformer layers.
• Fine‑tuned on a curated Twitter‑based emotion corpus (≈25 k labelled tweets).
• Supports PyTorch, TensorFlow, JAX and the safetensors format for fast loading.
• Ready‑to‑use with the Hugging Face pipeline("text‑classification") API.

Architecture highlights – DistilBERT removes the token‑type embeddings and the next‑sentence‑prediction head from the original BERT, replaces the traditional BERT’s 12‑layer stack with a 12‑layer “student” network that is trained via knowledge distillation. The result is a lean model that still benefits from the same self‑attention mechanisms and pre‑training on large‑scale corpora.

Intended use cases – Real‑time sentiment & emotion analysis in chatbots, social‑media monitoring, customer‑feedback pipelines, mental‑health support tools, and any application that needs a lightweight, high‑throughput emotion classifier.

Benchmark Performance

The most relevant benchmark for this model is the Emotion dataset, a multi‑class text‑classification task derived from Twitter. The model was evaluated on the official test split using the Hugging Face Trainer.

• Accuracy: 0.927 (verified)
• Macro Precision: 0.888 (verified)
• Macro Recall: 0.879 (verified)
• Macro F1‑Score: 0.883 (verified)
• Micro / Weighted Precision, Recall, F1: 0.927 (verified)
• Loss (cross‑entropy): 0.174 (verified)

These numbers place the model on par with full‑size BERT‑base fine‑tuned on the same data, while being considerably faster and lighter. For applications where latency and memory footprint matter, the trade‑off is highly favorable.

Hardware Requirements

• VRAM for inference: ~2 GB (FP32) or ~1 GB (FP16) is sufficient for a single sentence batch.
• Recommended GPU: Any modern GPU with ≥4 GB VRAM (e.g., NVIDIA RTX 2060, GTX 1660 Super). For high‑throughput batch processing, an 8 GB+ card (RTX 3060, A100, etc.) yields ~200 tokens / ms.
• CPU: 4‑core CPU with ≥8 GB RAM can run the model in real‑time for low‑volume traffic; multi‑core CPUs benefit from the torch‑set parallelism.
• Storage: The model files (weights + config + tokenizer) occupy ~256 MB when stored in the safetensors format.
• Performance characteristics: Inference latency is roughly 3‑5 ms per sentence on a 4 GB GPU, scaling linearly with batch size.

Use Cases

The model shines in any scenario where fast, on‑device or low‑latency emotion detection is required.

• Social‑media monitoring: Real‑time sentiment dashboards for brand reputation.
• Customer‑support chatbots: Detect frustration or joy to route conversations to human agents.
• Content moderation: Flag potentially harmful or abusive language based on emotional tone.
• Healthcare & mental‑health apps: Track emotional trends in user‑generated journal entries.
• Interactive entertainment: Adaptive storytelling that responds to player emotions.

Training Details

The model was fine‑tuned using the Hugging Face Trainer API with the following hyper‑parameters:

• Learning rate: 2e‑5
• Batch size (per GPU): 64
• Number of epochs: 8
• Optimizer: AdamW with weight decay 0.01
• Loss function: Cross‑entropy

The underlying dataset is the Emotion corpus, consisting of ~25 k English tweets annotated with six emotion labels. Training was performed on a single NVIDIA V100 (16 GB VRAM) for roughly 12 hours, consuming ~30 GB of GPU memory at peak (FP32). The final checkpoint was exported in safetensors for fast, zero‑copy loading.

Because the model is based on DistilBERT, it can be further fine‑tuned on domain‑specific data (e.g., product reviews, clinical notes) with the same hyper‑parameter template, making it a flexible starting point for many downstream tasks.

Related Papers

The core architecture originates from the DistilBERT paper:

• Sanh, V., Wolf, T., & Ruder, S. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108.

The fine‑tuning dataset is described in:

• Mohammad, S., & Kiritchenko, S. (2018). SemEval‑2018 Task 1: Affect in Tweets. arXiv:1808.07862.

Licensing Information

The repository tags the model with Apache‑2.0 licensing, even though the top‑level license field is listed as “unknown”. Apache‑2.0 is a permissive open‑source license that:

• Allows commercial, academic, and personal use without royalty.
• Permits modification, distribution, and inclusion in proprietary products.
• Requires preservation of the original copyright notice and a copy of the license.
• Mandates that any modified files carry a clear indication of changes.

In practice, you can embed this model in SaaS platforms, mobile apps, or on‑premise analytics pipelines, provided you retain the attribution to bhadresh‑savani and include the Apache‑2.0 license text in your distribution.