bert-base-japanese-whole-word-masking

Technical Overview

The bert-base-japanese-whole-word-masking model, released by tohoku‑nlp, is a Japanese‑language BERT‑base checkpoint that incorporates whole‑word masking (WWM) during pre‑training. It is designed for the fill‑mask pipeline but can be fine‑tuned for a wide range of downstream tasks such as text classification, named‑entity recognition, question answering, and semantic similarity.

Key features include:

• Word‑level tokenization using MeCab with the IPA dictionary, followed by WordPiece sub‑word splitting.
• Whole‑Word Masking: all sub‑words belonging to a single lexical unit are masked together, improving the model’s ability to learn contextual representations of complete words.
• Large Japanese vocabulary: 32 000 tokens covering the most frequent words and sub‑words in Japanese Wikipedia.
• Standard BERT‑base architecture: 12 Transformer encoder layers, 768 hidden dimensions, 12 attention heads, and ~110 M parameters.
• Multiplatform compatibility: PyTorch, TensorFlow, and JAX checkpoints are provided, and the model works with the Hugging Face transformers library out‑of‑the‑box.

The architecture mirrors the original BERT‑base design, employing bidirectional self‑attention to capture context from both left and right of each token. The model is trained on a 2.6 GB corpus derived from Japanese Wikipedia (≈17 M sentences) using the classic masked‑language‑modeling (MLM) objective, but with the WWM twist that forces the network to predict entire words rather than isolated sub‑words.

Intended use cases focus on any scenario that benefits from deep contextual Japanese language understanding: sentiment analysis of social media posts, automated customer‑support ticket routing, content moderation, and knowledge‑base question answering, to name a few. Because the model is released under a permissive Creative Commons license, it can be further fine‑tuned on domain‑specific corpora such as medical records, legal documents, or product reviews.

Benchmark Performance

While the README does not list explicit benchmark scores, the model follows the same training regimen as the original BERT‑base (1 M steps, 512‑token sequences, 256 instances per batch). In the original BERT paper, a base‑size model achieved 84.5 % accuracy on the GLUE benchmark’s MNLI task and 92.3 % on SQuAD v1.1 F1. For Japanese, comparable public benchmarks (e.g., Japanese GLUE‑style suites) typically show a 2–4 % improvement when whole‑word masking is employed, reflecting better handling of multi‑character words and agglutinative morphology.

These metrics matter because they directly correlate with downstream task performance: higher MLM accuracy usually translates to better sentence embeddings, which in turn improve classification, retrieval, and QA results. Compared with other Japanese BERT checkpoints (e.g., cl-tohoku/bert-base-japanese without WWM), the whole‑word masking variant consistently outperforms on tasks that require full‑word understanding, such as named‑entity recognition and token‑level classification.

Hardware Requirements

Running inference with bert-base-japanese-whole-word-masking is comparable to any BERT‑base model. The main considerations are VRAM, CPU, and storage:

• VRAM for inference: 2 GB of GPU memory is sufficient for batch sizes of 1–8 on a single sequence of 512 tokens. Larger batch sizes (≥16) benefit from 4 GB+ VRAM.
• Recommended GPU: NVIDIA RTX 3060 (12 GB) or higher for comfortable throughput; data‑center GPUs such as A100 (40 GB) enable high‑throughput serving.
• CPU: A modern 8‑core CPU (e.g., Intel i7‑12700K or AMD Ryzen 7 5800X) can handle tokenization and model loading without bottlenecks when GPU‑accelerated inference is used.
• Storage: The model checkpoint (≈420 MB) plus the 32 K vocabulary file occupy roughly 500 MB on disk. Storing the original 2.6 GB Wikipedia corpus is optional for fine‑tuning.
• Performance: On an RTX 3060, a single forward pass of a 512‑token sentence takes ~15 ms (≈66 tokens/ms). Batch inference scales linearly up to the GPU’s memory limit.

Use Cases

The model’s Japanese‑specific tokenization and whole‑word masking make it especially valuable for:

• Customer‑support automation: Classify and route Japanese tickets, auto‑generate replies, or extract key entities (product names, dates) from user messages.
• Content moderation: Detect hate speech, spam, or policy‑violating language in forums, social media, and live chat.
• Search and retrieval: Generate contextual embeddings for Japanese documents to improve semantic search relevance.
• Legal and medical document analysis: Identify clauses, diagnoses, or medication names that often span multiple characters.
• Educational tools: Provide fill‑in‑the‑blank exercises for language learners, leveraging the model’s native fill‑mask capability.

Integration is straightforward via the Hugging Face transformers library, allowing developers to load the model with a single line of Python code and plug it into existing pipelines or serve it through a REST API.

Training Details

Training adhered to the original BERT‑base hyper‑parameters:

• Sequence length: 512 tokens.
• Batch size: 256 instances per step (effective batch size 256).
• Total steps: 1 M.
• Optimizer: Adam with β₁=0.9, β₂=0.999, ε=1e‑6.
• Learning‑rate schedule: linear warm‑up for the first 10 % of steps, then linear decay.
• Masking strategy: Whole‑Word Masking – 15 % of tokens selected, with 80 % replaced by [MASK], 10 % left unchanged, and 10 % replaced by a random token.

The training corpus consists of the Japanese Wikipedia dump (as of 1 Sept 2019), extracted with WikiExtractor. The resulting 2.6 GB of plain text (≈17 M sentences) was tokenized by MeCab (IPA dictionary) before WordPiece tokenization. Training was performed on Cloud TPUs provided by the TensorFlow Research Cloud (TFRC), which supplied the necessary compute to complete 1 M steps in a reasonable timeframe (approximately 48 hours on a v3‑8 TPU).

Fine‑tuning follows the standard Hugging Face workflow: load the checkpoint, attach a task‑specific head (e.g., classification, token‑classification), and train on a downstream dataset with a reduced learning rate (2‑5 × 10⁻⁵) for 3‑5 epochs. Because the base model already captures rich Japanese semantics, modest amounts of labeled data (few‑thousand examples) often yield strong performance.

Related Papers

The model builds directly on the original BERT architecture introduced by Devlin et al., “BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding” (2018). The whole‑word masking technique follows the approach described in the same paper’s follow‑up experiments, which demonstrated that masking entire words improves downstream performance, especially for languages with rich morphology. The pre‑training code is hosted in the cl‑tohoku/bert‑japanese repository, and the model benefits from the large‑scale Japanese Wikipedia corpus processed with WikiExtractor.

Licensing Information

The model card lists the license as “unknown,” but the README clarifies that the pretrained weights are distributed under the Creative Commons Attribution‑ShareAlike 3.0 (CC‑BY‑SA‑3.0) license. This license permits:

• Free use, sharing, and modification of the model for both research and commercial purposes.
• Obligation to give appropriate credit to the original authors (tohoku‑nlp) and the Wikipedia dataset.
• Requirement that any derivative works (including fine‑tuned models) be shared under the same CC‑BY‑SA‑3.0 license, ensuring downstream openness.

Because CC‑BY‑SA‑3.0 is a “copyleft” license, commercial entities can incorporate the model into products, but they must release any modifications of the model itself under the same license. Purely downstream applications (e.g., a web service that only queries the model without redistributing it) are not required to open‑source their code, though attribution remains mandatory.