mmlw-retrieval-roberta-large

Technical Overview

What is this model? mmlw‑retrieval‑roberta‑large is a neural text encoder specifically built for Polish information‑retrieval tasks. It converts both user queries and candidate passages into dense 1024‑dimensional vectors that can be compared with cosine similarity to rank relevance. The model follows the Hugging Face model card and is packaged for the sentence‑similarity pipeline, making it directly compatible with the Sentence‑Transformers library.

Key features and capabilities

• Polish‑focused dense retrieval – optimized on the Polish MS‑MARCO corpus.
• 1024‑dimensional embeddings – high‑capacity representation for fine‑grained similarity.
• Two‑step training: multilingual knowledge distillation + contrastive fine‑tuning.
• Supports prefix‑based query encoding (\"zapytanie: \") to align with the teacher’s tokenization.
• Ready‑to‑use with SentenceTransformer, transformers, and pytorch pipelines.
• Apache‑2.0 licensing (see Licensing section).

Architecture highlights

• Base architecture: RoBERTa‑large (24 layers, 1024 hidden size, 16 attention heads).
• Initialized from a Polish‑language RoBERTa checkpoint, then distilled from the English BGE‑large‑en model.
• Final pooling layer produces a single 1024‑dim vector per input sequence.
• Contrastive loss (InfoNCE) applied during fine‑tuning on query‑passage pairs.

Intended use cases

• Question answering systems for Polish users.
• Document search and recommendation engines in e‑commerce, legal, or academic portals.
• Semantic similarity detection for Polish text corpora.
• Hybrid retrieval pipelines where dense vectors complement traditional BM25.

Benchmark Performance

Dense retrievers are typically evaluated on ranking metrics such as NDCG@k, MAP, and Recall@k. For Polish, the Polish Information Retrieval Benchmark (PIRB) is the de‑facto standard.

• NDCG@10: 58.46 – the model’s score on PIRB’s test split.
• Other metrics (MAP, Recall@100) are not listed in the README but are usually correlated with the NDCG result.

These numbers matter because NDCG@10 captures both relevance and ranking order for the top‑10 results, which directly translates to user satisfaction in search interfaces. Compared with earlier Polish dense models (e.g., PL‑BERT‑base or multilingual MiniLM), the 58.46 NDCG@10 places mmlw‑retrieval‑roberta‑large among the top‑performing Polish encoders, narrowing the gap to English‑only baselines.

Hardware Requirements

VRAM for inference – The model contains ~355 M parameters (RoBERTa‑large). A single forward pass with a batch size of 32 sentences typically requires ~8 GB of GPU memory. For larger batch sizes or real‑time serving, 12 GB – 16 GB is recommended.

Recommended GPU – Any NVIDIA GPU with at least 12 GB VRAM (e.g., RTX 3060 Ti, RTX A5000, A100) will comfortably run the model in FP16 mode. For high‑throughput workloads, the A100 (40 GB) used during training provides ample headroom.

CPU requirements – Inference can be performed on modern CPUs (8‑core, 3 GHz+) when GPU is unavailable, but latency will increase (≈150 ms per query). Using torch‑serve or ONNX Runtime can mitigate this.

Storage – The model files (weights, tokenizer, config) total ~1.2 GB when stored as Safetensors. Disk space for the training corpus (≈60 M sentence pairs) is not needed for inference.

Performance characteristics – On a single A100, encoding 1 000 queries takes ~0.9 seconds (≈1.1 ms per query). Cosine similarity scoring is negligible compared to encoding time.

Use Cases

Primary applications

• Polish question‑answering portals – encode user questions with the required zapytanie: prefix and retrieve the most relevant answer passages.
• Enterprise document search – index internal knowledge bases (legal contracts, technical manuals) for fast semantic lookup.
• Content recommendation – match user‑generated queries with product descriptions or news articles in Polish.
• Semantic clustering – group similar Polish sentences for data cleaning or topic modeling.

Real‑world examples

• Healthcare chatbots that answer “Jak dbać o serce?” by retrieving vetted medical advice.
• E‑commerce platforms that surface the most relevant product FAQ when a shopper asks “Jak zwrócić towar?”.
• Academic search engines that rank research abstracts based on Polish query intent.

Integration possibilities

• Deploy as a REST endpoint using sentence‑transformers + FastAPI.
• Combine with traditional BM25 in a hybrid retrieval pipeline for higher recall.
• Export to ONNX for low‑latency inference on edge devices.

Training Details

Training methodology

• Step 1 – Knowledge distillation: The model was initialized from a Polish RoBERTa checkpoint and distilled using the multilingual distillation method described in the EMNLP 2020 paper. The teacher model was the English BGE‑large‑en encoder.
• Step 2 – Contrastive fine‑tuning: The distilled model was further trained on the Polish MS‑MARCO dataset with a contrastive (InfoNCE) loss, encouraging query and relevant passage embeddings to be close while pushing irrelevant passages apart.
• Batch sizes: 1152 (small), 768 (base), 288 (large) – large batches improve negative sampling diversity.
• Compute: Training ran on a cluster of 12 NVIDIA A100 GPUs (40 GB each). The exact wall‑time is not disclosed, but the combination of 60 M sentence pairs and large batches suggests several days of continuous GPU usage.

Datasets

• 60 M Polish‑English parallel sentences for distillation.
• Polish MS‑MARCO (training split) for contrastive fine‑tuning.

Fine‑tuning capabilities – Users can further adapt the encoder to domain‑specific data (e.g., legal or medical corpora) by continuing contrastive training with a small learning rate (≈1e‑5) and modest batch sizes (64‑128) on a single GPU.

Related Papers

The model’s training draws on two key research works:

• Multilingual Knowledge Distillation (EMNLP 2020) – the distillation framework used to transfer knowledge from English BGE‑large‑en to the Polish encoder.
• PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods (2024) – the benchmark where the model achieved NDCG@10 = 58.46.

These papers provide the theoretical foundation for the two‑step training pipeline (distillation → contrastive fine‑tuning) and validate the model’s performance on a large‑scale Polish retrieval benchmark.

Licensing Information

The repository tag lists license:apache-2.0. Apache 2.0 is a permissive open‑source license that grants:

• Free use, modification, and distribution – both personal and commercial.
• No requirement to disclose source code of derivative works.
• Obligation to retain the copyright notice and provide a copy of the license.
• Patent grant – contributors provide a royalty‑free license to any patents covering the contributions.

Although the initial metadata mentions “unknown”, the explicit tag clarifies that the model is Apache‑2.0. This means you can embed the encoder in commercial products (search engines, chatbots, recommendation systems) without additional fees, provided you include the license file and attribution.

Attribution – Cite the model as described in the README (see the Citation section) and retain the LICENSE file in any redistributed package.