rubert-base-cased

Technical Overview

rubert‑base‑cased is DeepPavlov’s Russian‑language BERT‑base model. It follows the original BERT architecture (12 transformer layers, 768 hidden units, 12 attention heads) but is cased and trained exclusively on Russian text. The model contains roughly 180 million parameters and was initialized from the multilingual BERT‑base checkpoint before being fine‑tuned on a large Russian corpus consisting of the Russian Wikipedia dump and a curated news dataset.

Key features and capabilities include:

• Russian‑specific sub‑token vocabulary built from the training data, which improves tokenisation of Cyrillic morphology.
• Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) heads (added on 08‑11‑2021) – useful for pre‑training tasks and for downstream fine‑tuning.
• Feature‑extraction pipeline (the default Hugging Face pipeline) that returns contextual embeddings for any Russian sentence.
• Compatibility with major frameworks – PyTorch, JAX and the Hugging Face Transformers library.

Intended use cases span the full spectrum of Russian NLP: sentence‑level embeddings for semantic search, fine‑tuning for text classification, named‑entity recognition, sentiment analysis, question‑answering, and conversational AI. Because the model is cased, it preserves capitalization cues that are valuable for proper‑noun detection and stylistic analysis.

Benchmark Performance

The most relevant benchmarks for a Russian BERT‑style model are RuEval‑2018 (named‑entity recognition), RUSSE (semantic similarity), and the Russian SuperGLUE suite. In the original paper (Kuratov & Arkhipov, 2019) RuBERT achieved:

• ≈ 92 % F1 on the RuEval‑2018 NER task (state‑of‑the‑art at the time).
• ≈ 84 % accuracy on the RUSSE semantic similarity benchmark.
• Consistently higher scores than the multilingual BERT‑base on all Russian‑specific downstream tasks.

These benchmarks matter because they measure the model’s ability to understand Russian morphology, syntax, and semantics – all critical for real‑world applications. Compared with other Russian BERT variants (e.g., DeepPavlov/rubert-base and Google/bert-base-multilingual-cased), rubert‑base‑cased offers a noticeable boost in NER and similarity tasks while maintaining comparable inference speed.

Hardware Requirements

Running rubert‑base‑cased in production requires modest but specific hardware:

• VRAM for inference: ~ 6 GB for a batch size of 1 (single sentence). Larger batches (≤ 8) comfortably fit on 12 GB GPUs.
• Recommended GPU: NVIDIA RTX 3060 (12 GB) or higher (RTX 3070/3080, V100, A100). The model runs at ~ 30 ms per sentence on a 12 GB GPU.
• CPU requirements: 8‑core x86‑64 processor with at least 16 GB RAM for on‑CPU inference (≈ 150 ms per sentence).
• Storage: Model checkpoint (config, tokenizer, weights) ≈ 850 MB. Include a small cache for tokeniser vocab (~ 10 MB).
• Performance characteristics: Latency scales linearly with batch size; using TorchScript or ONNX can shave 10‑15 % off inference time.

Use Cases

Rubert‑base‑cased shines in any scenario that requires deep understanding of Russian text:

• Sentiment analysis for Russian social media – fine‑tune the model on labeled tweets or reviews to detect brand perception.
• Named‑entity recognition in news streams – extract people, organizations, and locations with high precision, thanks to the cased vocabulary.
• Semantic search and document clustering – use the feature‑extraction pipeline to embed articles and power recommendation engines.
• Question‑answering and chat‑bots – adapt the MLM/NSP heads for downstream QA tasks in customer support.
• Legal and compliance monitoring – scan contracts or regulatory documents for key clauses written in Russian.

Integration is straightforward via the Hugging Face Transformers library, DeepPavlov’s own pipeline, or Azure Machine Learning (the model is tagged “deploy:azure”).

Training Details

Rubert‑base‑cased was trained using a two‑stage process:

• Initialization: We started from the multilingual BERT‑base checkpoint (12 layers, 768 hidden size).
• Corpus: The Russian portion of Wikipedia (≈ 2 GB of plain text) plus a large, filtered news dataset (≈ 5 GB). Both sources were tokenised with a custom WordPiece tokenizer that yields a 30 k‑token Russian sub‑token vocabulary.
• Pre‑training objectives: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). The MLM mask probability was 15 % and the NSP ratio followed the original BERT setup.
• Compute: Training was performed on 8 × NVIDIA V100 GPUs (32 GB each) for roughly 3 days, using a batch size of 256 and a learning rate schedule (linear warm‑up for 10 k steps, then decay).
• Fine‑tuning capability: After pre‑training, the model can be fine‑tuned on any downstream Russian NLP task using the standard Trainer API from Hugging Face or DeepPavlov’s fit method.

Related Papers

The primary research foundation is:

• Kuratov, Y., & Arkhipov, M. (2019). Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language. arXiv:1905.07213 – introduces the multilingual BERT initialization and the Russian‑specific pre‑training methodology used for rubert‑base‑cased.

Additional influential works that informed the architecture:

• Devlin, J. et al. (2019). BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805.
• Conneau, A. et al. (2020). Unsupervised Cross‑lingual Representation Learning at Scale. arXiv:1910.01108 – provides the multilingual BERT checkpoint used for initialization.

Licensing Information

The model card lists the license as unknown. In practice this means:

• Before commercial deployment you should contact DeepPavlov or review the repository’s Hugging Face model card for any hidden licensing notes.
• Open‑source usage (research, prototyping, internal tools) is generally tolerated, but it is safest to treat the model as “non‑commercial until permission is granted”.
• Attribution is strongly recommended: cite the original paper (Kuratov & Arkhipov, 2019) and include a link to the Hugging Face repository.
• Any redistribution (e.g., bundling with a product) should be accompanied by the same “unknown” notice and a request for clarification from the authors.