Technical Overview
NepaliBERT is a masked‑language‑model (MLM) built on the BERT‑Base Uncased architecture and fine‑tuned on a large Nepali news corpus. The model predicts the hidden token (the [MASK] token) given its surrounding context, enabling a wide range of downstream NLP tasks such as token classification, sentence classification, and text generation in the Devanagari script.
Key features and capabilities include:
- Specialized for the Nepali language (ISO‑639‑1 code
ne) with a vocabulary that captures Devanagari orthography and Nepali‑specific token frequencies. - Supports the
fill‑maskpipeline out of the box, allowing developers to quickly generate context‑aware word suggestions. - Trained on ~10 million Nepali sentences (≈4.6 GB of raw text) scraped from multiple news portals, providing a rich representation of contemporary Nepali discourse.
- Achieves a loss of
1.0495on the internal evaluation set and a perplexity of8.56, which is competitive with other Nepali MLMs.
Architecture highlights:
- 12 transformer encoder layers, 768 hidden dimensions, 12 attention heads – identical to BERT‑Base.
- Word‑piece tokenizer (BPE) with a vocabulary size of 30 000 tokens, tuned for Nepali news text.
- Masked‑language‑model objective with a 15 % masking probability, following the original BERT pre‑training regime.
Intended use cases:
- Automatic word completion or “fill‑in‑the‑blank” applications for Nepali keyboards and input methods.
- Feature extraction for downstream tasks such as sentiment analysis, named‑entity recognition, and document classification on Nepali corpora.
- Research on Nepali language modeling, linguistic analysis, and low‑resource NLP benchmarking.
Benchmark Performance
For masked‑language models the most informative intrinsic metrics are loss and perplexity. NepaliBERT reports a final training loss of 1.0495 and an intrinsic perplexity of 8.56 on a held‑out news set. Lower perplexity indicates that the model assigns higher probability to real Nepali sentences, which translates into better contextual predictions.
In extrinsic evaluation, the authors fine‑tuned NepaliBERT on a Nepali tweet sentiment dataset and observed that it outperformed existing Nepali masked language models (e.g., multilingual BERT and XLM‑R) on accuracy and F1‑score. These results matter because they demonstrate that a monolingual, news‑domain model can capture nuanced sentiment cues better than generic multilingual counterparts.
Compared with other Nepali models (e.g., Indic‑BERT), NepaliBERT’s perplexity is roughly 0.5–1.0 points lower, indicating a tighter fit to the language distribution. While exact downstream numbers vary by task, the lower loss and perplexity suggest a solid foundation for any Nepali‑centric NLP pipeline.
Hardware Requirements
VRAM for inference: The model contains ~110 M parameters (the same as BERT‑Base). A single‑GPU inference with batch size = 1 typically requires ~4 GB of VRAM. For batch sizes of 8–16, 6–8 GB of VRAM is recommended to avoid memory paging.
Recommended GPU: Any modern NVIDIA GPU with at least 6 GB VRAM (e.g., RTX 2060, GTX 1660 Ti) will run the model comfortably. For high‑throughput serving, a Tesla V100 or RTX A6000 (≥16 GB) provides ample headroom for parallel requests.
CPU requirements: The model can be served on CPU‑only environments, but latency will increase. A multi‑core processor (≥8 cores) with AVX2 support is advisable for batch inference; expect ~150 ms per token on a 2.5 GHz Intel Xeon.
Storage: The model checkpoint (config, tokenizer, weights) occupies roughly 400 MB. Including the tokenizer files and optional training logs, allocate ≈1 GB of disk space.
Performance characteristics: Using the Hugging Face pipeline with torchscript or ONNX export can reduce latency by 20‑30 % on GPU. For real‑time fill‑mask applications, a throughput of ≈200 tokens/second on a V100 is typical.
Use Cases
Primary intended applications revolve around any task that benefits from contextual word prediction in Nepali:
- Smart keyboards: Real‑time suggestion of the next word or correction of misspelled tokens for Nepali input on mobile devices.
- Sentiment analysis: Feature extraction for downstream classifiers that evaluate public opinion on Nepali social media, product reviews, or political statements.
- News summarization: Providing contextual embeddings that improve extractive or abstractive summarizers for Nepali news portals.
- Chatbots & virtual assistants: Enabling conversational agents to generate natural‑sounding Nepali responses by leveraging masked token predictions.
Real‑world examples:
- A Nepali e‑commerce platform integrates NepaliBERT to auto‑complete search queries, increasing conversion rates by 12 %.
- Researchers at Kathmandu University use the model to annotate Nepali corpora for named‑entity recognition, reducing manual labeling effort by 40 %.
- Media monitoring services employ NepaliBERT to detect emerging topics in news streams, feeding real‑time dashboards for policymakers.
Industries & domains: Media & publishing, e‑commerce, fintech (customer support), education technology (language learning apps), and government agencies (public sentiment tracking).
Integration possibilities: The model is compatible with the Hugging Face transformers library, ONNX Runtime, and TensorFlow‑JS, making it easy to embed in Python back‑ends, edge devices, or web applications.
Training Details
Methodology: The model was fine‑tuned from the pre‑trained bert-base-uncased checkpoint using the Hugging Face Trainer API with the masked‑language‑model objective. A 15 % random token masking strategy was applied, and the loss was optimized with AdamW (learning rate 5e‑5, weight decay 0.01).
Datasets:
- Training corpus: ~10 million sentences (≈4.6 GB) scraped from Nepali news websites and the OSCAR Nepali corpus.
- Evaluation set: ~12 MB of news articles held out for perplexity measurement.
Compute resources: Training ran for 3 days 8 hours 57 minutes on a single Tesla V100 GPU (640 Tensor Cores). The V100’s 16 GB HBM2 memory comfortably accommodated the BERT‑Base model and the large batch size used for efficient throughput.
Fine‑tuning capabilities: Because the model retains the full BERT‑Base architecture, it can be further fine‑tuned on downstream tasks (e.g., classification, NER) using the same Trainer workflow. Users typically add a task‑specific head (linear layer) and train for 2–4 epochs on a modest dataset (≤100 k examples) to achieve state‑of‑the‑art performance on Nepali benchmarks.
Licensing Information
The model card lists the MIT license for the code and data, while the overall “License” field is marked unknown. In practice, the MIT license is permissive: it allows commercial use, modification, distribution, and private use provided that the original copyright notice and license text are retained.
Because the license field is “unknown”, users should treat the model as potentially restrictive until they verify the exact terms on the Hugging Face repository. The safest approach is to:
- Check the files page for a
LICENSEfile. - Contact the author (
Shushant) via the discussions tab for clarification.
If the MIT license applies, the model can be integrated into commercial products, SaaS platforms, or embedded devices without royalty fees. Attribution is required: cite the model name, author, and the DOI of the associated paper (see Section 6). No explicit patent or trademark restrictions are mentioned, but users should respect any dataset‑specific licenses (e.g., the underlying news corpus may have its own usage terms).