phobert-base

Technical Overview

PhoBERT‑Base is a monolingual, transformer‑based language model pre‑trained for Vietnamese (code vi). It follows the RoBERTa training recipe, which refines the original BERT objective by using a larger batch size, longer training, and a more aggressive masking strategy. The model is available on the Hugging Face Hub under the identifier vinai/phobert-base.

Key features and capabilities include:

• 12 transformer layers (base size), 768 hidden dimensions, 12 attention heads.
• Optimized for the fill‑mask pipeline, making it ideal for masked‑language‑model inference and downstream fine‑tuning.
• Pre‑trained on a massive Vietnamese corpus (≈ 20 GB of cleaned text from Wikipedia, news, and web crawls).
• Supports PyTorch, TensorFlow, and JAX via the 🤗 Transformers library.
• MIT‑compatible licensing (see licensing section) and ready for deployment on Azure, AWS, or on‑premise servers.

Architecture highlights:

• Based on the RoBERTa‑base architecture (12 × Transformer encoder blocks).
• Uses Byte‑Pair Encoding (BPE) with a 50 k token vocabulary tuned for Vietnamese morphology.
• Trained with a 15 % token‑masking rate, dynamic masking per batch, and a cosine learning‑rate schedule.
• No next‑sentence‑prediction head – only the masked‑language‑model head, which simplifies fine‑tuning for classification, tagging, and NER.

Intended use cases cover a wide range of Vietnamese NLP tasks:

• Masked token prediction (fill‑mask) for data augmentation and interactive applications.
• Fine‑tuning for part‑of‑speech tagging, dependency parsing, named‑entity recognition, and natural‑language inference.
• Feature extraction for downstream classifiers, semantic search, or recommendation systems.
• Rapid prototyping of Vietnamese chatbots and virtual assistants.

Benchmark Performance

PhoBERT‑Base was evaluated on four core Vietnamese NLP benchmarks, achieving new state‑of‑the‑art scores at the time of publication (EMNLP‑2020 Findings). The most relevant metrics are:

• Part‑of‑Speech (POS) tagging: 96.5 % accuracy.
• Dependency parsing: 92.1 % UAS (Unlabeled Attachment Score).
• Named‑Entity Recognition (NER): 90.3 % F1.
• Natural Language Inference (NLI): 84.7 % accuracy.

These benchmarks matter because they test the model’s understanding of syntax, semantics, and contextual reasoning—core abilities for any Vietnamese language system. Compared with multilingual models such as mBERT or XLM‑R, PhoBERT‑Base consistently outperforms by 3–7 % absolute on each task, demonstrating the advantage of a dedicated, large‑scale monolingual pre‑training corpus.

Hardware Requirements

For inference with the fill‑mask pipeline, PhoBERT‑Base requires roughly 4 GB of GPU VRAM (the model’s 110 M parameters occupy ~2 GB, leaving head for tensors and batch overhead). A single‑GPU setup with at least 8 GB VRAM (e.g., NVIDIA Tesla T4, RTX 3060, or A100 40 GB) provides smooth latency below 50 ms per request for batch size = 1.

Recommended GPU specifications:

• CUDA ≥ 11.1, cuDNN ≥ 8.0.
• 8 GB VRAM minimum; 16 GB+ for larger batches or simultaneous multi‑task serving.
• Tensor‑core enabled GPUs (e.g., NVIDIA A100, RTX 3090) accelerate mixed‑precision (FP16) inference.

CPU requirements:

• 8‑core modern CPU (Intel i7‑9700K or AMD Ryzen 7 3700X) for preprocessing and tokenization.
• At least 16 GB RAM to hold the tokenizer vocab and intermediate buffers.

Storage needs:

• Model checkpoint size ≈ 420 MB (PyTorch .bin file).
• Tokenizer files (vocab and merges) add another ~30 MB.
• Overall, ≈ 500 MB of disk space is sufficient.

Performance scales linearly with batch size and can be further improved using ONNX or TorchScript export.

Use Cases

PhoBERT‑Base shines in any scenario that demands a deep understanding of Vietnamese text. Typical applications include:

• Masked‑language‑model services – auto‑completion, spelling correction, and contextual word suggestion in Vietnamese keyboards.
• Fine‑tuned downstream tasks – POS tagging, NER, and dependency parsing for linguistic research or information extraction pipelines.
• Chatbots and virtual assistants – powering intent detection and response generation for customer‑service bots serving Vietnamese‑speaking users.
• Semantic search & recommendation – embedding generation for document retrieval, news recommendation, or e‑commerce product matching.
• Academic and industry research – a baseline model for Vietnamese language understanding experiments.

Because the model is fully compatible with the 🤗 Transformers library, integration is straightforward in Python, JavaScript (via transformers.js), or even mobile frameworks that support ONNX Runtime.

Training Details

PhoBERT‑Base follows the RoBERTa‑style pre‑training pipeline:

• Corpus: A cleaned, deduplicated Vietnamese text collection (~20 GB) assembled from Wikipedia, news outlets, and web crawls.
• Tokenizer: Byte‑Pair Encoding with a 50 k vocabulary, specifically tuned to Vietnamese word‑piece patterns.
• Training objective: Masked Language Modeling (MLM) with a 15 % token masking rate, dynamic per‑batch masking.
• Optimizer: AdamW with a learning‑rate of 1e‑4, linear warm‑up for the first 10 k steps, then cosine decay.
• Compute: Trained on 8 × NVIDIA V100 GPUs (32 GB each) for ~2 weeks, totaling ~300 k GPU‑hours.
• Batch size: 8 k tokens per GPU (effective batch ≈ 64 k).
• Fine‑tuning: The model can be fine‑tuned on downstream tasks using the standard 🤗 Transformers Trainer API, typically requiring 2–4 GB GPU memory for batch size = 16.

All training scripts are open‑source on the VinAI GitHub repository, enabling reproducibility and custom extensions.

Related Papers

The primary research foundation for PhoBERT‑Base is the EMNLP‑2020 Findings article:

• PhoBERT: Pre‑trained language models for Vietnamese – Dat Quoc Nguyen & Anh Tuan Nguyen, 2020.

The model also builds on the RoBERTa methodology (Liu et al., 2019) and the original BERT architecture (Devlin et al., 2018). These works introduced the masked‑language‑model objective and the large‑scale transformer encoder that PhoBERT adapts for Vietnamese.

Licensing Information

The repository’s license field lists MIT, yet the model card also tags the license as “unknown”. In practice, the underlying code and model weights are released under the MIT License, which is a permissive open‑source license.

What the MIT license permits:

• Free use, copy, modification, and distribution of the model and source code.
• Commercial exploitation – you may embed PhoBERT‑Base in paid products, SaaS platforms, or on‑device applications.
• No requirement to release derivative works under the same license.

Potential restrictions:

• Although the license is permissive, the “unknown” tag suggests you should verify the exact terms on the original GitHub repository before large commercial deployment.
• Attribution is required – the original authors (Dat Quoc Nguyen & Anh Tuan Nguyen) must be credited in documentation or product acknowledgments.

For academic publications, the authors request that you cite their EMNLP‑2020 paper (see the “Related Papers” section).