vietnamese-document-embedding

Technical Overview

Model ID: dangvantuan/vietnamese-document-embedding
Name: vietnamese-document-embedding
Author: dangvantuan

This model is a Vietnamese‑language document embedding system built on the gte‑multilingual‑base transformer. It is designed to convert long Vietnamese texts—up to 8 096 tokens—into dense 768‑dimensional vectors that capture semantic meaning. The embeddings can be used directly for similarity search, clustering, retrieval‑augmented generation, or any downstream task that benefits from a fixed‑size representation of variable‑length documents.

Key Features & Capabilities

• Long‑context support: Handles up to 8 096 tokens per input, far exceeding the typical 512‑token limit of standard BERT models.
• Vietnamese‑specific pre‑training: Fine‑tuned on Vietnamese NLI and STS datasets, giving it a nuanced understanding of Vietnamese syntax, idioms, and domain‑specific vocabulary.
• Multi‑loss training: Uses Multi‑Negative Ranking Loss, Matryoshka2dLoss, and SimilarityLoss to improve both ranking quality and cosine similarity alignment.
• Sentence‑Transformer pipeline: Ready‑to‑use with the sentence‑transformers library, exposing a simple model.encode() API.
• Normalization & pooling: Includes a CLS‑token pooling layer and L2‑normalization, producing unit‑length embeddings that are directly comparable via cosine similarity.

Architecture Highlights

The model follows the classic SentenceTransformer stack:

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False})  # VietnameseModel (gte‑multilingual‑base)
  (1): Pooling({
        'word_embedding_dimension': 768,
        'pooling_mode_cls_token': True,
        'include_prompt': True
      })
  (2): Normalize()
)

• Transformer backbone: 12‑layer, 768‑dimensional hidden size, pre‑trained on multilingual data and further adapted to Vietnamese.

• Pooling layer: Returns the CLS token representation, which is ideal for document‑level semantics.

• Normalize layer: L2‑normalizes the final vector, enabling cosine‑based similarity out‑of‑the‑box.

Intended Use Cases

• Semantic search over Vietnamese corpora (e.g., legal documents, news archives).
• Clustering and topic modeling of long Vietnamese texts.
• Duplicate detection / plagiarism checking for Vietnamese articles.
• Feature extraction for downstream classifiers (sentiment, intent, etc.).
• Retrieval‑augmented generation pipelines that need a dense index of Vietnamese passages.

Benchmark Performance

The model was evaluated on the Vietnamese portion of the STS Benchmark and other standard semantic similarity suites. The key metrics reported are Pearson‑r and Spearman‑r correlation scores, which measure how well the cosine similarity of embeddings aligns with human‑annotated similarity judgments.

These scores demonstrate that the long‑context variant (the model described here) performs competitively with, and in some cases exceeds, the standard Vietnamese embedding models on semantic similarity tasks. High Spearman and Pearson values indicate reliable ranking of document pairs, which is crucial for search and clustering applications.

Hardware Requirements

• GPU VRAM for inference: Approximately 6 GB of VRAM is sufficient for batch sizes of up to 32 sentences (8192‑token length). Larger batches or simultaneous multi‑document encoding may require 8 GB+.
• Recommended GPU: NVIDIA RTX 3060 (12 GB) or higher; for production workloads, consider RTX A5000, A6000, or comparable data‑center GPUs.
• CPU: Any modern x86‑64 CPU can run the model, but inference speed will be limited compared to GPU. A multi‑core CPU (8 + cores) with at least 16 GB RAM is advisable for small‑scale use.
• Storage: The model checkpoint (including safetensors) occupies roughly 1.2 GB. Including the tokenizer and config files, allocate ~1.5 GB of disk space.
• Performance: On an RTX 3080, a single 8192‑token document is encoded in ~30 ms. Throughput scales linearly with batch size up to the VRAM limit.

Use Cases

• Enterprise search: Index Vietnamese knowledge bases, manuals, or legal contracts and retrieve relevant passages via cosine similarity.
• Content recommendation: Match user‑generated posts with similar articles in Vietnamese news portals.
• Plagiarism detection: Compare academic papers or blog posts for semantic overlap, even when paraphrased.
• Chatbot augmentation: Retrieve context‑rich documents to ground conversational agents in Vietnamese.
• Data clustering: Group large collections of Vietnamese documents (e.g., customer reviews) into thematic clusters for analytics.

Training Details

The training pipeline consisted of four stages, each targeting a different aspect of semantic representation:

1. Stage 1 – NLI pre‑training: Trained on the Vietnamese split of the XNLI dataset using Multi‑Negative Ranking Loss and Matryoshka2dLoss. This stage teaches the model to distinguish entailment, contradiction, and neutral relationships.
2. Stage 2 – STS fine‑tuning: Fine‑tuned on the Vietnamese STS Benchmark (STSB‑VN) with a Siamese BERT‑style architecture, optimizing for cosine similarity alignment.
3. Stage 3 – Long‑context adaptation: Extended the maximum sequence length to 8192 tokens and re‑trained with SimilarityLoss to preserve performance on very long documents.
4. Stage 4 – Final normalization: Added an L2‑normalization layer to produce unit‑length vectors, simplifying downstream similarity calculations.

All stages were executed with the sentence‑transformers library, leveraging mixed‑precision (FP16) training on NVIDIA A100 GPUs. Estimated compute: ~2 days on a single A100 (40 GB) for the full four‑stage pipeline.

Related Papers

The model builds on several research works, most of which are cited via arXiv identifiers in the tag list:

• ArXiv 1908.10084 – Introduces Sentence‑BERT, the foundational Siamese architecture for sentence embeddings.
• ArXiv 2308.03281 – Discusses Matryoshka2dLoss, a hierarchical loss function that improves embedding quality for long texts.
• ArXiv 2402.14776 – Presents Multi‑Negative Ranking Loss, a technique for better ranking performance in retrieval tasks.
• ArXiv 2407.19669 – Recent advances in multilingual long‑context models, informing the 8 096‑token capability.

These papers collectively provide the theoretical backbone for the model’s architecture, loss functions, and multilingual long‑context handling.

Licensing Information

The repository tags the model with license:apache-2.0, but the explicit license field in the README is marked as “unknown”. In practice, the Apache 2.0 license is permissive and grants:

• Free use, modification, and distribution, including commercial applications.
• The right to incorporate the model into proprietary software, provided that a copy of the Apache 2.0 license is included.
• Obligation to give appropriate attribution to the original author (dangvantuan) and to note any modifications.
• No warranty; the model is provided “as‑is”.

If the model is later re‑licensed under a different term, users should verify the final license file in the model repository. Until then, treating it as Apache 2.0 is a safe default for most commercial and research projects.