rubert-tiny2

Technical Overview

What is this model? cointegrated/rubert‑tiny2 is a compact, Russian‑only BERT‑based encoder designed to generate high‑quality sentence embeddings. It is the successor of the original rubert‑tiny model and focuses on delivering strong semantic similarity scores while keeping the footprint small enough for edge‑device deployment.

Key features and capabilities

• Very small size (≈ 12 M parameters) – “tiny” BERT that fits comfortably on a single GPU with < 4 GB VRAM.
• Extended vocabulary of 83 828 tokens, covering modern Russian orthography, slang, and domain‑specific terms.
• Longer context window – up to 2 048 tokens per sequence, a four‑fold increase over the previous 512‑token limit.
• Sentence‑level embeddings (312‑dimensional) that approximate the performance of the large multilingual LaBSE model.
• Segment embeddings tuned on a Natural Language Inference (NLI) task, giving the model a better sense of sentence polarity and entailment.
• Ready‑to‑use with sentence‑transformers, transformers, and VLLM‑optimized variants.
• Supports mask‑fill (masked‑language‑model) tasks, making it a versatile pre‑training backbone.

Architecture highlights – The model follows the classic BERT encoder stack (12 transformer layers, 12 attention heads, hidden size 312) but with a reduced hidden dimension to keep the parameter count low. The tokenization relies on SentencePiece, providing sub‑word coverage for Russian morphology. Because the model is trained exclusively on Russian corpora, the embedding space is densely packed for Cyrillic scripts, which improves downstream similarity tasks.

Intended use cases – The primary purpose is to produce sentence‑level embeddings for:

• Semantic search and retrieval in Russian text collections.
• Nearest‑neighbor (K‑NN) classification of short messages, tweets, or support tickets.
• Clustering and topic modeling on Russian corpora.
• Feature extraction for downstream classifiers (e.g., sentiment analysis, intent detection).
• Masked‑language‑model inference (fill‑mask) for quick prototyping.

Benchmark Performance

For a tiny encoder, the most relevant benchmarks are sentence similarity (e.g., STS‑B) and semantic retrieval metrics (Recall@k). The README highlights that the new version brings the embeddings closer to LaBSE, a large multilingual model, while staying under 12 M parameters.

Key metrics (as reported)

• Embedding dimension: 312.
• Sequence length support: 2 048 tokens.
• Vocabulary size: 83 828 tokens.
• Similarity to LaBSE: improved cosine‑similarity scores on Russian STS datasets (exact numbers not disclosed).

Why these benchmarks matter – Sentence similarity directly reflects how well the model captures semantic nuance, which is critical for search, clustering, and K‑NN classification. A longer context window reduces the need for aggressive truncation, preserving meaning in longer Russian documents.

Comparison to similar models – Compared with the original rubert‑tiny (512‑token limit, 29 564‑token vocab), rubert‑tiny2 delivers:

• ~2× better coverage of Russian lexicon.
• ~4× longer context handling.
• Higher semantic similarity scores, approaching LaBSE while using < 5 % of its parameters.

Hardware Requirements

VRAM for inference – The model’s checkpoint (≈ 200 MB in safetensors format) plus the tokenization cache comfortably fits into a 2 GB GPU memory budget. However, to run batch inference efficiently (e.g., batch size = 32), a GPU with at least 4 GB VRAM is recommended.

Recommended GPU specifications

• Any modern NVIDIA GPU with ≥ 4 GB VRAM (e.g., GTX 1650, RTX 2060, Tesla T4).
• CUDA ≥ 11.0 and the latest torch/ transformers libraries.
• For VLLM‑optimized inference, a GPU with fast tensor cores (e.g., RTX 3080) can achieve > 200 tokens/sec.

CPU requirements – On CPU‑only machines, inference is slower but still feasible for low‑throughput workloads. A recent 8‑core CPU (e.g., AMD Ryzen 7 5800X) with ≥ 16 GB RAM provides acceptable latency (≈ 150 ms per sentence).

Storage needs – The model files (weights + tokenizer) occupy roughly 250 MB. Including the sentence‑transformers wrapper and optional VLLM‑optimized files, total storage is under 500 MB.

Performance characteristics – Typical latency on a single RTX 2070:

• Single‑sentence embedding: ~ 5 ms.
• Batch of 32 sentences: ~ 30 ms (GPU) / ~ 400 ms (CPU).

Use Cases

Primary intended applications

• Semantic search – Index large Russian corpora (news archives, legal documents) and retrieve relevant passages via cosine similarity.
• Customer support automation – Cluster incoming tickets to route them to appropriate agents.
• Content moderation – Detect near‑duplicate or plagiarized Russian text.
• Recommendation systems – Match user‑generated queries with product descriptions in Russian e‑commerce.
• Low‑resource devices – Deploy on edge servers or Raspberry Pi‑class hardware for on‑device inference.

Real‑world examples

• A Russian social‑media monitoring tool that clusters trending topics in real time.
• A bilingual chatbot that uses the Russian embeddings for intent detection while the English side relies on a separate encoder.
• Academic research pipelines that need fast sentence similarity for large‑scale corpus analysis.

Integration possibilities – The model works out‑of‑the‑box with:

• Python sentence‑transformers library – one‑line model.encode() calls.
• Hugging Face transformers API – easy to wrap in FastAPI or Flask services.
• VLLM – high‑throughput serving on GPU clusters.
• ONNX Runtime – for CPU‑only or mobile deployment.

Training Details

Training methodology – The model was pre‑trained on a large Russian corpus using the masked‑language‑model (MLM) objective, followed by a fine‑tuning stage on a Natural Language Inference (NLI) dataset to learn meaningful segment embeddings. This two‑step approach mirrors the strategy used for LaBSE and other multilingual sentence‑embedding models.

Datasets

• Russian Wikipedia, news feeds, and web‑crawled text (≈ 30 GB of raw tokens).
• Open‑source NLI datasets translated into Russian (e.g., Russian SNLI, RUSSE).

Compute requirements – Training was performed on a mixed‑precision (FP16) setup using 8 × NVIDIA V100 GPUs for roughly 2 weeks. The final checkpoint size reflects the reduced hidden dimension and the larger vocabulary.

Fine‑tuning capabilities – Because the model is released as a standard transformers checkpoint, you can:

• Fine‑tune on domain‑specific NLI or paraphrase data.
• Apply adapter layers for lightweight task‑specific adaptation.
• Use the sentence‑transformers trainer to further improve similarity on your own corpus.

Related Papers

The README does not list explicit arXiv or DOI links, but the model builds on several well‑known works:

• BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding – the foundational architecture.
• LaBSE: Language‑agnostic BERT Sentence Embedding – the multilingual benchmark that rubert‑tiny2 aims to approximate for Russian.
• Sentence‑Transformers: Multilingual Sentence Embeddings using Siamese BERT Networks – the framework used for inference.

These papers provide the theoretical underpinnings for the transformer encoder, the contrastive training objectives, and the sentence‑level pooling strategy employed in rubert‑tiny2.

Licensing Information

The model card lists the license as MIT, while the tag metadata shows “license: unknown”. The MIT license is permissive, allowing unrestricted use, modification, distribution, and commercial exploitation, provided that the original copyright notice and license text are retained.

Commercial usage – Under MIT, you may embed the model in SaaS products, on‑premise services, or mobile applications without paying royalties. The only legal requirement is to include the MIT license file in your distribution.

Restrictions & requirements

• No warranty – the model is provided “as is”.
• No trademark usage – you cannot claim the model as your own.
• Attribution – keep the original author credit (“cointegrated”) in documentation or UI.

If you distribute a fine‑tuned version, the same MIT terms apply, but you may also attach an additional license of your own to the derived work, as long as the original MIT notice remains intact.