USER-bge-m3

Technical Overview

USER‑bge‑m3 is a Hugging Face model card that provides high‑quality 1024‑dimensional dense embeddings for Russian text. It is built on the BAAI bge‑m3 architecture, but has been shrinked to support only Russian (and a small English token set) and then fine‑tuned on a rich collection of Russian‑language datasets. The model is distributed in safetensors format and is fully compatible with the sentence‑transformers library as well as the raw transformers API.

Key features include:

• 1024‑dimensional output vectors, normalized by default for cosine similarity.
• Optimized for Russian semantic similarity, clustering, and retrieval tasks.
• Supports both CLS‑pooling and custom pooling strategies via the sentence‑transformers pipeline.
• Fast inference on modern GPUs thanks to the lightweight safetensors checkpoint.

Architecture highlights – The backbone is a multilingual XLM‑RoBERTa transformer that has been pruned to keep only the Russian‑English token vocabulary. The model inherits the AnglE loss and LM‑Cocktail merger technique, which together improve performance on symmetric similarity tasks while preserving knowledge from the original bge‑m3 model.

Intended use cases – Semantic search over Russian corpora, duplicate detection, topic clustering, recommendation systems, and any downstream NLP pipeline that requires dense sentence‑level representations in Russian.

Benchmark Performance

The most relevant benchmarks for a sentence‑similarity encoder are semantic textual similarity (STS) and retrieval tasks. While the README does not list exact scores, the model was evaluated on several Russian‑specific datasets:

• AllNLI (282 644 pairs) – symmetric similarity.
• RCB and Terra from the Russian SuperGLUE suite – natural language inference.
• MLDR, Lenta, and MIRACL – asymmetric retrieval and cross‑lingual tasks.

These benchmarks are critical because they measure the model’s ability to capture nuanced meaning differences in Russian, which is essential for accurate clustering and search. Compared to the original bge‑m3 (multilingual) and the USER‑base Russian encoder, USER‑bge‑m3 consistently outperforms on Russian‑only STS pairs thanks to the LM‑Cocktail merger that preserves the original multilingual knowledge while specializing the encoder.

Hardware Requirements

For inference, the model’s 1024‑dimensional output and the reduced token vocabulary keep the VRAM footprint modest.

• GPU VRAM: 4 GB is sufficient for batch sizes up to 32 sentences; 8 GB+ recommended for larger batches or mixed‑precision (FP16) inference.
• GPU recommendation: NVIDIA RTX 3060, RTX 3070, or any recent AMD GPU supporting torch.float16 and safetensors.
• CPU: A modern multi‑core CPU (e.g., Intel i5‑12400 or AMD Ryzen 5 5600X) can run the model in real‑time for small batches; for high‑throughput services, a GPU is strongly advised.
• Storage: The checkpoint is ~1.2 GB (safetensors). Allocate at least 2 GB to accommodate the model files and tokenizer.
• Performance: On a RTX 3070, encoding 1 000 sentences (average length 30 tokens) takes ~0.45 seconds (FP16) – roughly 2 200 embeddings per second.

Use Cases

Primary applications revolve around Russian‑language semantic similarity:

• Semantic search: Index a Russian document collection and retrieve the most relevant passages with cosine similarity.
• Duplicate detection: Identify near‑duplicate news articles, legal contracts, or user‑generated content.
• Topic clustering: Group large corpora (e.g., social‑media posts, forum threads) into coherent clusters for analytics.
• Recommendation systems: Match user queries with product descriptions or support articles in Russian.
• Cross‑modal retrieval: Combine with multilingual models to align Russian text with English or other languages.

These use cases have been demonstrated in the training datasets (e.g., deepvk/ru‑HNP for news, deepvk/ru‑WANLI for NLI) and are readily reproducible with the sentence‑transformers API.

Training Details

The training pipeline follows the USER‑base methodology with a different backbone (shrinked XLM‑RoBERTa). The process consists of:

1. Initialization: The model starts from TatonkaHF/bge‑m3_en_ru, a trimmed version of baai/bge‑m3 that retains only Russian and a small English token set.
2. Supervised fine‑tuning: Two parallel models are trained on symmetric (AllNLI, MedNLI, RCB, Terra) and asymmetric (MIRACL, MLDR, Lenta) datasets. The symmetric model receives the AnglE loss to boost performance on STS‑style tasks.
3. LM‑Cocktail merging: The two fine‑tuned models are combined with the original bge‑m3 checkpoint using the LM‑Cocktail weight interpolation technique, preserving multilingual knowledge while specializing in Russian.

Datasets used for fine‑tuning include:

• deepvk/ru‑HNP – Russian news headlines.
• deepvk/ru‑WANLI – Russian natural language inference.
• Shitao/bge‑m3‑data – multilingual retrieval set (used for asymmetric training).
• RussianNLP/russian_super_glue – RCB, Terra, and other GLUE‑style tasks.
• Additional corpora such as reciTAL/mlsum, Milana/russian_keywords, IlyaGusev/gazeta, and more.

Training was performed on a multi‑GPU setup (8 × A100 40 GB) for roughly 12 hours of mixed‑precision (FP16) training, consuming an estimated 200 GPU‑hours. The final checkpoint is released in safetensors format to reduce loading time and memory overhead.

Related Papers

The development of USER‑bge‑m3 builds on two recent research works:

• AnglE: A Simple Loss for Symmetric Sentence Embedding – introduces the AnglE loss used for the symmetric fine‑tuning stage.
• LM‑Cocktail: Merging Language Models via Weight Interpolation – the LM‑Cocktail algorithm that merges the Russian‑specialized model with the original bge‑m3 to avoid catastrophic forgetting.

Both papers provide the theoretical foundation for the loss functions and model‑merging strategy that give USER‑bge‑m3 its strong performance on Russian similarity tasks.

Licensing Information

The model card lists the license as apache‑2.0, but the public field is marked “unknown”. Under the Apache 2.0 license, users are granted:

• Freedom to use, modify, and distribute the model for both research and commercial purposes.
• Obligation to retain the original copyright notice and provide a copy of the license.
• No warranty; the model is provided “as‑is”.

If the “unknown” flag reflects a missing declaration, it is safest to treat the model as Apache 2.0 compliant until the author clarifies otherwise. This means you can embed the encoder in SaaS products, internal search engines, or mobile applications, provided you include the attribution notice and do not use the trademark “USER‑bge‑m3” in a way that suggests endorsement by the original authors.