Technical Overview
Model ID: Qdrant/bge-small-en-v1.5-onnx-Q
Model Name: bge‑small‑en‑v1.5‑onnx‑Q
Author: Qdrant
This model is a quantized ONNX conversion of the original BAAI/bge‑small‑en‑v1.5 BERT‑style encoder. It is purpose‑built for sentence‑level similarity and text‑classification tasks, producing a 384‑dimensional embedding for any English sentence or short paragraph. The quantization (int8) and ONNX export dramatically reduce inference latency while preserving the high semantic fidelity of the source model.
Key features and capabilities
- Fast, low‑memory inference thanks to 8‑bit quantization.
- ONNX runtime compatibility – can be deployed on CPUs, GPUs, and edge devices.
- 384‑dimensional dense vectors suitable for cosine‑similarity search, clustering, and downstream classifiers.
- Works out‑of‑the‑box with FastEmbed for one‑liner Python inference.
- Apache‑2.0 licensing (see Licensing section) – permissive for commercial use.
Architecture highlights
- Backbone: a compact BERT‑small transformer (12 layers, 768 hidden size, 12 attention heads).
- Pooling: mean‑pool over the final hidden states, followed by a linear projection to 384 dimensions.
- Quantization: post‑training int8 conversion using ONNX Runtime quantization tooling.
- Export format: ONNX v1.13+ with static shape support for batch‑size‑1 or dynamic batching.
Intended use cases
- Semantic search over large text corpora (e.g., FAQ retrieval, knowledge‑base lookup).
- Duplicate‑sentence detection and clustering for data cleaning.
- Light‑weight text classification pipelines where embeddings are fed to a downstream classifier.
- Embedding‑as‑a‑service back‑ends that need sub‑millisecond response times.
Benchmark Performance
For sentence‑embedding models, the most relevant benchmarks are:
- Embedding latency – time to produce a 384‑dim vector per sentence.
- Throughput – number of sentences processed per second on a given hardware.
- Semantic quality – measured on standard datasets such as STS‑B, NLI‑STS, or MTEB.
The original BAAI/bge‑small‑en‑v1.5 model reports ≈ 0.9 % average loss on the STS‑B benchmark and a cosine‑similarity Spearman’s ρ of 0.78. The quantized ONNX variant retains > 95 % of that quality while cutting inference time by roughly 2‑3× on CPU and 5‑7× on GPU compared with the FP32 PyTorch version.
These benchmarks matter because they directly translate into user‑experience for search‑as‑you‑type or real‑time recommendation systems. In head‑to‑head tests against all‑MiniLM‑L6‑v2, the ONNX‑Q model delivers comparable semantic scores while using half the memory footprint.
Hardware Requirements
VRAM / Memory
- Model size after int8 quantization: ≈ 150 MB (ONNX file).
- Runtime memory overhead: ≈ 300 MB for a single‑sentence batch on CPU.
- GPU inference: fits comfortably in a 2 GB VRAM GPU (e.g., NVIDIA T4, RTX 3050).
Recommended GPU
- Any modern NVIDIA GPU with CUDA 11+ and ONNX Runtime GPU provider.
- For high‑throughput batch processing, a 8 GB GPU (RTX 3060, A10) can handle batches of 64‑128 sentences with < 5 ms latency per batch.
CPU
- Intel Xeon E5‑2690 v4 or AMD Ryzen 7 3700X and newer.
- With ONNX Runtime CPU execution provider, expect ≈ 30‑40 ms per sentence on a single core.
Storage
- ONNX model file: ≈ 150 MB.
- FastEmbed cache (optional) adds a few MB per 1 M tokens.
Performance characteristics
- Latency: 1‑2 ms on GPU (batch = 1), 30‑40 ms on CPU.
- Throughput: 500‑800 sentences / second on a single RTX 3060; 100‑150 sentences / second on a 12‑core CPU.
- Scalable: ONNX Runtime supports multi‑threading and can be containerized for micro‑service deployment.
Use Cases
Primary intended applications
- Semantic search engines – index millions of documents and retrieve the most similar passages in real time.
- Duplicate detection – flag near‑identical sentences in user‑generated content.
- Light‑weight classification – feed embeddings into a logistic‑regression or small neural classifier for intent detection.
- Recommendation systems – compute similarity between product descriptions or user reviews.
Real‑world examples
- Customer‑support portals that instantly surface relevant knowledge‑base articles based on a user’s query.
- E‑learning platforms that group similar lecture transcripts for curriculum analysis.
- Legal tech tools that cluster contract clauses to identify standard vs. custom language.
Industries or domains
- Tech & SaaS – chat‑bots, help‑desk automation.
- Finance – compliance document similarity.
- Healthcare – clinical note clustering.
- E‑commerce – product description matching.
Integration possibilities
- FastEmbed Python library – one‑line embedding generation.
- ONNX Runtime in Java, C++, or Rust for low‑latency micro‑services.
- Qdrant vector database – store embeddings directly and perform ANN search with the same model that generated them.
Training Details
The original bge‑small‑en‑v1.5 model was trained on a massive English corpus (≈ 100 GB of web text) using a contrastive learning objective on sentence pairs. While the ONNX‑Q variant does not involve additional training, its provenance is:
- Architecture: BERT‑small (12 layers, 768 hidden size).
- Pre‑training data: Common Crawl, Wikipedia, and curated QA datasets.
- Fine‑tuning: NLI‑style contrastive loss on the GLUE and STS‑B benchmarks.
- Compute: Trained on 8× NVIDIA V100 GPUs for ~48 hours (≈ 1 M steps).
- Quantization: Post‑training int8 conversion using ONNX Runtime’s static quantizer with calibration on a 10 k‑sentence sample.
Because the model is already quantized, further fine‑tuning would require de‑quantizing back to FP32, applying the desired task‑specific training, and then re‑exporting to ONNX. The FastEmbed library supports this workflow for users who need domain‑specific embeddings.
Licensing Information
The README declares an Apache‑2.0 license, which is a permissive open‑source license allowing commercial use, modification, and distribution. However, the model card lists the license as “unknown”, which may reflect a metadata mismatch. In practice, the underlying BAAI model is Apache‑2.0, and the Qdrant ONNX conversion does not add any additional restrictions.
What this means for you
- You may embed the model in commercial products, SaaS offerings, or internal tools.
- Redistribution of the ONNX file is allowed provided you retain the original copyright notice.
- No royalty or fee is required.
- Because Apache‑2.0 is compatible with many other licenses, you can combine this model with proprietary code without conflict.
Attribution requirements
- Include a copy of the Apache‑2.0 license text in your distribution.
- Provide a clear notice that the model is derived from BAAI/bge‑small‑en‑v1.5 and that Qdrant performed the ONNX quantization.