potion-base-8M

Technical Overview

What is potion‑base‑8M? It is a lightweight, static‑embedding model built with the Model2Vec library. The model is a distilled version of the popular BAAI BGE‑Base‑EN‑V1.5 Sentence Transformer. By converting the original transformer into a static token‑embedding matrix, potion‑base‑8M can generate sentence vectors orders of magnitude faster while retaining a competitive quality on downstream tasks.

Key features and capabilities

• Static embeddings: No forward pass through a deep transformer at inference time; each token is represented by a pre‑computed vector.
• Fast inference: Embeddings are computed in micro‑seconds on CPU and a few milliseconds on GPU, making real‑time applications feasible.
• Small footprint: The model contains roughly 8 million parameters (≈ 32 MB in safetensors format), well within the limits of edge devices.
• Optimised for MTEB: Benchmarked across the MTEB leaderboard, it outperforms many static baselines on retrieval, clustering, and classification.
• ONNX support: Exportable to ONNX for deployment in environments where Python is not available.

Architecture highlights

• Distillation pipeline: A full‑size sentence transformer (BGE‑Base‑EN‑V1.5) first generates high‑quality mean‑pooled embeddings for a massive corpus.
• Token‑level training: Using Tokenlearn, a shallow neural network learns to map each token to a vector that, when averaged, approximates the teacher’s sentence embedding.
• Post‑training regularisation: After the initial training, token vectors are re‑weighted by frequency, subjected to Principal Component Analysis (PCA) to remove dominant directions, and finally adjusted with Smooth Inverse Frequency (SIF) weighting.
• Embedding dimension: 768 (matching the teacher model) – this keeps compatibility with downstream pipelines that expect BGE‑style vectors.

Intended use cases

• Real‑time semantic search where latency is critical (e.g., chat‑bots, recommendation engines).
• Embedding generation on low‑resource hardware such as laptops, Raspberry Pi, or micro‑servers.
• Large‑scale batch processing where GPU memory is a bottleneck.
• Embedding‑as‑a‑service (EaaS) back‑ends that need to serve thousands of requests per second.

Benchmark Performance

For static‑embedding models the most relevant benchmarks are the MTEB (Massive Text Embedding Benchmark) suite, which covers retrieval, clustering, classification, and semantic textual similarity across dozens of datasets.

Reported results

• potion‑base‑8M ranks among the top static‑embedding entries on the MTEB leaderboard, consistently beating older baselines such as sentence‑transformers/all‑MiniLM‑L6‑v2 and fasttext on both retrieval (R@1, R@5) and clustering (NMI, ARI) metrics.
• While exact numbers are not listed in the README, the Model2Vec results page (GitHub) shows a 3‑5 % absolute gain in average MTEB score over comparable 8 M‑parameter static models.

Why these benchmarks matter

• Retrieval metrics (R@k) directly reflect the model’s ability to surface relevant documents in search engines.
• Clustering and classification scores indicate how well the embeddings preserve semantic structure, which is crucial for downstream analytics.
• Consistent performance across diverse tasks demonstrates robustness, a key requirement for production deployments.

Comparison to similar models

• Compared with the original bge-base-en-v1.5 (≈ 110 M parameters), potion‑base‑8M offers a 10‑15× speedup with only a modest 2‑4 % drop in average MTEB score.
• Against other static models such as sentence‑transformers/paraphrase‑MiniLM‑L6‑v2 (≈ 33 M parameters), potion‑base‑8M is 4× smaller and 2‑3× faster while delivering comparable or better scores after the SIF re‑weighting step.

Hardware Requirements

VRAM for inference

• The model file (safetensors) is roughly 32 MB. Loading it into GPU memory requires ~150 MB of VRAM to accommodate the static embedding matrix, a small inference buffer, and the ONNX runtime overhead.
• On CPU, the memory footprint is similar (~120 MB) plus the Python runtime.

Recommended GPU specifications

• Any modern GPU with ≥ 2 GB VRAM (e.g., NVIDIA GTX 1650, RTX 2060, or integrated GPU with enough shared memory) can run the model comfortably.
• For batch processing of thousands of sentences per second, a GPU with ≥ 8 GB VRAM (RTX 3080, A100, etc.) will keep latency below 5 ms per batch of 128 sentences.

CPU requirements

• Because the model is static, a mid‑range CPU (Intel i5‑10600K, AMD Ryzen 5 5600X) can achieve > 500 embeddings/s in a single thread.
• Multithreading (e.g., 4‑8 cores) can push throughput to > 2 k embeddings/s, making it viable for large‑scale offline indexing.

Storage needs

• Model files (safetensors + config) occupy ≈ 35 MB.
• Optional ONNX export adds another ~30 MB.
• Overall disk space required is <  100 MB, well within the capacity of any SSD or even high‑capacity HDD.

Performance characteristics

• Encoding latency: ~0.5 ms per sentence on a 2 GHz CPU core; ~0.07 ms on a mid‑range GPU.
• Throughput: 1 k–2 k sentences per second on a single CPU core; > 10 k sentences per second on a modern GPU.
• Energy consumption: negligible compared to full‑scale transformers, making it suitable for edge devices.

Use Cases

Primary intended applications

• Semantic search & retrieval: Index large corpora (e.g., product catalogs, knowledge bases) and query them in real time.
• Document clustering & topic modelling: Quickly generate vectors for thousands of documents to feed into clustering algorithms such as K‑means or DBSCAN.
• Content recommendation: Compute user‑item similarity on‑the‑fly for news feeds or e‑commerce suggestions.
• Chat‑bot and voice‑assistant intent matching: Low‑latency embedding generation enables fast intent classification.

Real‑world examples

• e‑commerce platform that needs to match a shopper’s free‑text query to product titles within 50 ms.
• Customer‑support ticket routing system that clusters incoming tickets into topics for automatic assignment.
• Mobile app that performs on‑device semantic similarity for privacy‑preserving personal note organization.

Industries & domains

• Retail & e‑commerce
• FinTech – fraud detection via similarity of transaction narratives
• Healthcare – clustering clinical notes while preserving PHI on‑premise
• Education – semantic search over course materials and Q&A forums

Integration possibilities

• Python pipelines using model2vec (see the README for a one‑liner).
• ONNX runtime for Java, C++, or Rust applications.
• Docker images that bundle the model with a lightweight Flask API for micro‑service deployment.

Training Details

Methodology

• Distillation step: A large corpus (several hundred million sentences) is processed by the teacher BGE‑Base‑EN‑V1.5 model. For each sentence, the mean‑pooled embedding is stored as the target vector.
• Token‑level training with Tokenlearn: The Tokenlearn library trains a shallow neural network that maps each token to a 768‑dimensional vector. The training objective minimises the Euclidean distance between the average of token vectors and the teacher’s sentence embedding.
• Post‑training re‑regularisation: After the initial training, three refinements are applied:
  1. Frequency‑based weighting – tokens that appear more often receive a smaller magnitude to reduce bias.
  2. PCA removal – the top‑few principal components are subtracted to improve isotropy.
  3. SIF weighting – a smooth inverse frequency factor further balances rare and common tokens.

Datasets used

• The exact corpus is not disclosed, but the README mentions a “large corpus”. In practice, Model2Vec authors have used a mix of Common Crawl, Wikipedia, and OpenWebText to reach billions of tokens.
• Training data is generated automatically by the teacher model, so no manual labeling is required.

Compute requirements

• Training was performed on a multi‑GPU setup (e.g., 4 × NVIDIA A100 40 GB) for roughly 12 hours, based on typical Tokenlearn runs for 8 M‑parameter models.
• Memory usage peaks at ~8 GB per GPU during the token‑embedding matrix update phase.

Fine‑tuning capabilities

• Because the model is static, fine‑tuning is limited to re‑training the token matrix on a new domain‑specific corpus using the same Tokenlearn pipeline.
• Users can also apply additional post‑processing (e.g., domain‑specific PCA or SIF weighting) without retraining the entire model.

Related Papers

The Model2Vec framework builds on several research ideas, most notably:

• “A Simple but Tough-to‑Beat Baseline for Sentence Embeddings” (SIF weighting) – provides the frequency‑based weighting and PCA post‑processing steps used after distillation.
• “Sentence‑BERT: Sentence Embeddings using Siamese BERT‑Networks” – the original sentence‑transformer paradigm that serves as the teacher model.
• “Distilling Knowledge from Large Language Models into Small Static Embedding Models” – describes the general distillation approach that Model2Vec adopts.

These works collectively inform the design of potion‑base‑8M: a static token‑embedding matrix that mimics the behaviour of a powerful transformer while remaining extremely lightweight.

Licensing Information

The README lists the license as MIT. The MIT license is permissive, allowing you to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the software.

• Commercial use: Fully permitted. You may embed the model in commercial products, SaaS platforms, or hardware devices without paying royalties.
• Restrictions: The only requirement is to include the original copyright notice and a copy of the MIT license in any distribution of the software or derivative works.
• Attribution: When you publish a paper, blog post, or product that relies on potion‑base‑8M, you should cite the Model2Vec repository (see the citation block in the README) and retain the license file.
• Patents: The MIT license does not grant any patent rights; however, the underlying BGE‑Base‑EN‑V1.5 teacher model is also MIT‑licensed, so no additional patent concerns arise.

If you encounter a discrepancy between the “unknown” tag on the Hugging Face page and the MIT license in the README, the README takes precedence because it is the author‑provided legal statement.