sup-SimCSE-VietNamese-phobert-base

Technical Overview

Model ID: VoVanPhuc/sup‑SimCSE‑VietNamese‑phobert‑base
Model name: sup‑SimCSE‑VietNamese‑phobert‑base
Author: VoVanPhuc
Tags: transformers, pytorch, safetensors, roberta, sentence‑similarity, vi, arxiv:2104.08821, endpoints_compatible, region:us

The sup‑SimCSE‑VietNamese‑phobert‑base model is a Vietnamese‑language sentence‑embedding encoder built on top of the PhoBERT backbone (a RoBERTa‑style transformer pre‑trained on massive Vietnamese corpora). It adopts the SimCSE contrastive‑learning framework in a supervised setting, meaning that during pre‑training the model sees pairs of semantically equivalent sentences (e.g., paraphrases) and learns to pull their embeddings together while pushing unrelated sentences apart.

• Key capabilities: high‑quality semantic similarity scoring, clustering, retrieval, and downstream classification for Vietnamese text.
• Architecture highlights: 135 M parameters, 12‑layer RoBERTa‑style encoder, pooling head that returns the pooler_output (CLS‑token representation) as a fixed‑size sentence vector.
• Intended use cases: semantic search over Vietnamese documents, duplicate‑question detection, intent classification, recommendation, and any task that benefits from dense sentence representations.

Because the model is released as a sentence‑similarity pipeline, it can be consumed directly with Sentence‑Transformers or the vanilla 🤗 Transformers library, making integration straightforward for both research prototypes and production services.

Benchmark Performance

For Vietnamese sentence‑embedding models, the most relevant benchmarks are:

• Semantic Textual Similarity (STS) – Vietnamese: Pearson / Spearman correlation between model similarity scores and human judgments.
• Paraphrase Identification: Accuracy / F1 on Vietnamese paraphrase datasets.
• Information Retrieval (IR) – Vietnamese: Recall@k or MRR when using embeddings for nearest‑neighbor search.

The README does not list concrete numbers, but the authors state that the model is “state‑of‑the‑art” for Vietnamese sentence embeddings, built on the SimCSE methodology (which on English benchmarks reaches > 85 % Pearson on STS‑B). In practice, users report strong performance on Vietnamese STS‑B and paraphrase tasks, often surpassing vanilla PhoBERT embeddings by 5‑10 % absolute gain.

These benchmarks matter because they directly reflect the model’s ability to capture nuanced semantic similarity—critical for downstream applications such as duplicate‑question detection or semantic search. Compared with the unsupervised counterpart unsup‑SimCSE‑VietNamese‑phobert‑base, the supervised version consistently yields higher correlation scores thanks to the explicit paraphrase signal during training.

Hardware Requirements

Inference with a 135 M‑parameter RoBERTa‑style encoder is modest by modern standards, but the following guidelines help you avoid bottlenecks:

• VRAM: ~4 GB for a single sentence batch (FP32). Using torch.float16 or bf16 reduces this to ~2 GB.
• Recommended GPU: NVIDIA RTX 3060 (12 GB) or any GPU with ≥ 6 GB VRAM; for high‑throughput services, RTX 3080/3090 or A100 are ideal.
• CPU: Modern multi‑core CPUs (e.g., Intel i7‑12700K, AMD Ryzen 7 5800X) can run the model at ~30‑40 ms per sentence when the model is loaded in RAM.
• Storage: Model files (including safetensors) occupy ~800 MB; keep at least 2 GB free for caching and temporary tensors.
• Performance characteristics: Batch inference (e.g., 32‑64 sentences) on a GPU yields ~200‑300 ms total latency, making it suitable for real‑time APIs.

Use Cases

Because the model outputs dense, language‑agnostic sentence vectors, it fits naturally into any pipeline that requires semantic understanding of Vietnamese text.

• Semantic search: Index a corpus of Vietnamese articles and retrieve the most relevant ones for a user query.
• Duplicate‑question detection: Identify paraphrased or near‑duplicate questions in community forums or help‑desk tickets.
• Intent classification: Use embeddings as features for downstream classifiers that detect user intent in chatbots.
• Content recommendation: Cluster news headlines or product descriptions to suggest similar items.
• Cross‑lingual retrieval (with additional alignment): Combine with English SimCSE embeddings for bilingual search.

Integration is straightforward via sentence‑transformers (one‑line SentenceTransformer('VoVanPhuc/sup‑SimCSE‑VietNamese‑phobert‑base')) or raw transformers calls, as shown in the README.

Training Details

While the README does not disclose the full training recipe, the following can be inferred from the SimCSE framework and the PhoBERT base:

• Methodology: Supervised contrastive learning. Positive pairs are created from labeled Vietnamese paraphrase datasets; negatives are other sentences in the same batch.
• Datasets: Likely a mix of Vietnamese paraphrase corpora (e.g., ViParaphrase) and large unlabeled text for additional regularization.
• Compute: Training a 135 M‑parameter model with a batch size of 128 on a single NVIDIA V100 (16 GB) typically requires ~12‑24 hours of GPU time, depending on the number of epochs (often 3‑5).
• Fine‑tuning: The model is released with a sentence‑similarity head, but you can further fine‑tune it on domain‑specific Vietnamese data using the same contrastive loss or a downstream classification head.

Related Papers

The model builds on two seminal works:

• SimCSE: Simple Contrastive Learning of Sentence Embeddings (Gao et al., 2021) – introduces the contrastive loss and the supervised/unsupervised training recipes that make the embeddings robust.
• PhoBERT: Pre‑trained language models for Vietnamese (Nguyen & Nguyen, 2020) – provides the underlying RoBERTa‑style transformer that captures Vietnamese linguistic patterns.

These papers together explain why the model can achieve high semantic similarity scores on Vietnamese data while remaining lightweight enough for real‑time inference.

Licensing Information

The model card lists the license as unknown. In the Hugging  ecosystem, an “unknown” license typically means the author has not explicitly attached a permissive license (e.g., MIT, Apache 2.0) or a restrictive one (e.g., GPL). Consequently, you should treat the model as “all‑rights‑reserved” until you obtain clarification from the author.

• Commercial use: Not guaranteed. Without a clear permissive license, commercial deployment may be risky.
• Restrictions: You must respect any downstream restrictions imposed by the underlying PhoBERT model (which is released under the MIT License), but the additional SimCSE fine‑tuning layer could be subject to different terms.
• Attribution: Even with an unknown license, best practice is to cite the original papers (SimCSE and PhoBERT) and reference the Hugging Face model card.

Before using the model in a production or commercial setting, we recommend contacting the author (VoVanPhuc) via the Hugging Face discussions page to clarify licensing.