ko-sroberta-multitask

Technical Overview

ko‑sroberta‑multitask is a Korean‑language sentence‑transformer built on top of a RoBERTa‑base encoder (768‑dimensional hidden size). The model maps a single sentence or an entire paragraph into a dense vector that captures its semantic meaning, making it ideal for downstream tasks such as semantic search, clustering, duplicate detection, and any application that requires fast similarity comparison.

Key features & capabilities:

• Language: Korean (ko) – trained on Korean corpora and fine‑tuned on Korean NLI (KorNLI) and semantic similarity (KorSTS) datasets.
• Embedding dimension: 768 – compatible with most vector databases (FAISS, Milvus, Pinecone, etc.).
• Pooling: Mean‑pooling over token embeddings (the default for Sentence‑Transformers), which respects the attention mask for accurate averaging.
• Multi‑task training: Simultaneous optimization on NLI (ranking) and similarity (cosine) losses, yielding robust representations across both tasks.
• Framework compatibility: Works out‑of‑the‑box with sentence‑transformers, 🤗 Transformers, PyTorch, and TensorFlow (via the tf tag).

Architecture highlights:

• Base encoder: RobertaModel with a maximum sequence length of 128 tokens, no lower‑casing.
• Pooling layer: Pooling configured for mean‑token pooling only (CLS token disabled, max‑pooling disabled).
• Training losses: MultipleNegativesRankingLoss (scale = 20, cosine similarity) followed by CosineSimilarityLoss for fine‑tuning.

Intended use cases:

• Semantic search over Korean document collections.
• Clustering of Korean sentences for topic modeling or deduplication.
• Real‑time similarity scoring in chat‑bots, recommendation engines, and QA systems.
• Feature extraction for downstream Korean NLP pipelines (e.g., classification, ranking).

Benchmark Performance

The most relevant benchmark for Korean sentence embeddings is KorSTS – a semantic textual similarity dataset derived from Korean NLI pairs. The model was evaluated on the KorSTS test split after multi‑task training on KorNLI and KorSTS.

• Cosine Pearson: 84.77
• Cosine Spearman: 85.60
• Euclidean Pearson: 83.71
• Euclidean Spearman: 84.40
• Manhattan Pearson: 83.70
• Manhattan Spearman: 84.38
• Dot‑product Pearson: 82.42
• Dot‑product Spearman: 82.33

These scores place ko‑sroberta‑multitask among the top Korean sentence‑embedding models on KorSTS, surpassing earlier monolingual BERT‑based baselines by 5‑7 percentage points. High Pearson and Spearman correlations indicate that the embedding space preserves both linear and rank‑order similarity, which is crucial for reliable semantic search and clustering.

Hardware Requirements

The model contains ~125 M parameters (the standard RoBERTa‑base size). Inference is lightweight compared to generative models, but the following hardware guidelines ensure smooth operation:

• VRAM for inference: 4 GB GPU memory is sufficient for batch sizes up to 32 sentences (128‑token sequences). Larger batches or longer sequences benefit from 8 GB or more.
• Recommended GPU: NVIDIA RTX 3060 (12 GB) or higher; for production‑scale indexing, consider RTX A5000 (24 GB) or A100 (40 GB).
• CPU: Any modern x86‑64 CPU (Intel i5 10th gen or AMD Ryzen 5 3600) can run the model via the sentence-transformers CPU backend, though latency will be ~2‑3× slower than GPU.
• Storage: Model files total ~500 MB (config, tokenizer, and weights). Keep an extra 200 MB for the tokenizer vocab and cache.
• Performance: On a RTX 3060, encoding 1 000 Korean sentences (128 tokens each) takes roughly 0.9 seconds (≈1 k sentences/s). CPU‑only speed is ~150 sentences/s.

Use Cases

Primary applications:

• Korean semantic search: Index Korean articles, FAQs, or product descriptions and retrieve the most relevant passages with cosine similarity.
• Duplicate detection: Identify near‑identical Korean sentences in large corpora (e.g., news feeds, user‑generated content).
• Clustering & topic modeling: Group similar Korean sentences for unsupervised analysis or to create labeled datasets.
• Feature extraction for classification: Use the 768‑dimensional embeddings as input features for downstream Korean classifiers (sentiment, intent, etc.).

Real‑world examples:

• Customer‑support chat‑bots that need to match user queries to a knowledge‑base of Korean FAQs.
• E‑commerce platforms that recommend similar Korean product descriptions.
• Legal document management systems that cluster similar Korean clauses for review.

The model integrates seamlessly with sentence‑transformers pipelines, FAISS indexes, or any vector‑search service that accepts 768‑dimensional float vectors.

Training Details

The model was trained in two stages using the sentence‑transformers library.

• Stage 1 – Ranking:
  • DataLoader: NoDuplicatesDataLoader with 8 885 Korean sentence pairs, batch size = 64.
  • Loss: MultipleNegativesRankingLoss (scale = 20, cosine similarity).
• Stage 2 – Similarity Fine‑tuning:
  • DataLoader: standard torch.utils.data.DataLoader with 719 samples, batch size = 8, random sampling.
  • Loss: CosineSimilarityLoss.
• Training hyper‑parameters:
  • Epochs: 5
  • Learning rate: 2 × 10⁻⁵ (AdamW)
  • Warm‑up steps: 360 (WarmupLinear scheduler)
  • Weight decay: 0.01
  • Gradient clipping (max norm): 1

The model uses a maximum sequence length of 128 tokens, which balances coverage of typical Korean sentences with memory efficiency. Fine‑tuning on custom Korean data is straightforward: replace the loss with CosineSimilarityLoss or MultipleNegativesRankingLoss, adjust the batch size, and run 2‑3 additional epochs.

Related Papers

The research foundation of ko‑sroberta‑multitask rests on two seminal works:

• Kornli and korsts: New benchmark datasets for Korean natural language understanding (2020) – provides the KorNLI and KorSTS datasets used for multi‑task training and evaluation.
• Sentence‑BERT: Sentence Embeddings using Siamese BERT‑Networks (2019) – introduces the sentence‑transformer architecture and pooling strategies that this model inherits.
• Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation (2020) – informs the multi‑task training approach, although the Korean model remains monolingual.

Licensing Information

The model card lists the license as unknown. In practice, this means the repository does not explicitly declare a permissive license such as MIT or Apache‑2.0. Users should treat the model as “all‑rights‑reserved” until clarification is obtained from the author (jhgan).

• Commercial use: Without a clear license, commercial deployment carries legal risk. It is advisable to contact the author for permission or to seek an alternative model with a known permissive license.
• Restrictions: Redistribution, modification, or integration into proprietary software may be prohibited unless explicitly granted.
• Attribution: Even under an unknown license, best practice is to cite the original authors (see “Citing & Authors” section) and include a link to the Hugging Face model card.
• Compliance: When using the model in a product, keep a record of the source URL and any correspondence with the author to demonstrate good‑faith compliance.