Technical Overview
Model ID: stelterlab/Mistral-Small-3.2-24B-Instruct-2506-FP8
Name: Mistral‑Small‑3.2‑24B‑Instruct‑2506‑FP8
Author: stelterlab
Base Model: mistralai/Mistral‑Small‑3.2‑24B‑Instruct‑2506
Mistral‑Small‑3.2‑24B‑Instruct‑2506‑FP8 is a quantized, 24‑billion‑parameter instruction‑tuned LLM that targets high‑quality text generation while drastically reducing memory footprint. The model is quantized to 8‑bit floating point (FP8, W8A8) using the llm‑compressor (v0.6.1.dev0+gc052d2ce.d20250624). This experimental quantization enables deployment on a single high‑end GPU without sacrificing the core capabilities of the original Mistral‑Small‑3.2 model.
Key capabilities include:
- Instruction following: Improved prompt adherence compared to the 3.1 predecessor.
- Reduced repetition: Halved the rate of infinite or looping generations.
- Robust function calling: More reliable parsing of tool‑call syntax.
- Multilingual support: Native handling of 27 languages (EN, FR, DE, ES, PT, IT, JA, KO, RU, ZH, AR, FA, ID, MS, NE, PL, RO, SR, SV, TR, UK, VI, HI, BN, etc.).
- Vision‑language pipeline: Tagged as
image-text-to-text, allowing image‑conditioned text generation via vLLM.
Architecturally, the model retains the transformer stack of Mistral‑Small‑3.2 (24 B parameters, 64‑layer decoder, 4096‑dimensional hidden states) but stores weights in compressed‑tensors and safetensors format for fast loading. The FP8 quantization reduces each weight to a single byte, cutting VRAM usage by ~4× while preserving the original model’s accuracy profile.
Intended use cases span chat assistants, code generation, multilingual content creation, and vision‑language tasks where low latency and modest hardware are required.
Benchmark Performance
The model’s performance is evaluated on a suite of text‑and‑vision benchmarks that are standard for instruction‑tuned LLMs. The most relevant metrics include Wildbench v2 (instruction following), Arena Hard v2 (chat quality), MMLU (knowledge), MATH (mathematical reasoning), and several vision‑language tests (MMMU, ChartQA, etc.).
- Instruction following (Wildbench v2): 65.33 % (vs. 55.6 % for 3.1).
- Chat quality (Arena Hard v2): 43.1 % (vs. 19.56 %).
- Internal IF accuracy: 84.78 % (vs. 82.75 %).
- Infinite generations: 1.29 % (2× reduction vs. 2.11 %).
- STEM (MMLU‑Pro 5‑shot CoT): 69.06 % (vs. 66.76 %).
- Code generation (MBPP Plus @5): 78.33 % (vs. 74.63 %).
- HumanEval Plus @5: 92.90 % (vs. 88.99 %).
- Vision (ChartQA): 87.4 % (vs. 86.24 %).
These benchmarks demonstrate that the FP8 quantized version retains or improves upon the original model’s strengths while offering a more efficient inference profile. The reductions in repetitive output and the gains in instruction fidelity make it especially suitable for production chatbots and tool‑calling pipelines.
Hardware Requirements
Because the model is stored in FP8 (W8A8) format, the VRAM demand drops dramatically compared to the full‑precision 24 B model. Typical requirements are:
- GPU VRAM: ~30 GB for a single‑GPU inference (e.g., NVIDIA A100 40 GB, RTX 4090 24 GB with tensor‑float‑16 off‑loading).
- GPU Compute: Ampere or newer architecture with support for FP8 kernels (NVIDIA Hopper or later) for optimal speed.
- CPU: Modern x86‑64 with at least 8 cores; CPU memory of 64 GB is recommended to hold the tokenizer and auxiliary data.
- Storage: Model files occupy ~45 GB (compressed‑tensors + safetensors). SSD NVMe is advised for fast loading.
- Inference Speed: Using vLLM with
--enable-auto-tool-choiceandtemperature=0.15yields ~30 tokens/s on a single A100.
Use Cases
Mistral‑Small‑3.2‑24B‑Instruct‑2506‑FP8 shines in scenarios where high‑quality language generation meets tight hardware budgets:
- Chatbots & virtual assistants: Precise instruction following and low repetition make it ideal for customer‑support bots.
- Code assistance: Strong performance on MBPP and HumanEval Plus enables autocomplete, bug‑fix suggestions, and unit‑test generation.
- Multilingual content creation: Supports 27 languages out‑of‑the‑box for translation, summarization, and content rewriting.
- Vision‑language applications: The
image-text-to-textpipeline tag allows image‑conditioned prompting (e.g., describing diagrams, extracting text from screenshots). - Tool‑calling & function execution: Robust function‑calling template makes it suitable for agents that need to invoke external APIs or run code.
Industries such as e‑commerce, education, software development, and media can embed the model into their platforms to deliver intelligent, low‑latency responses without a multi‑GPU cluster.
Training Details
The base model, Mistral‑Small‑3.2‑24B‑Instruct‑2506, was trained on a mixture of publicly available text corpora (the Crawl, Wikipedia, GitHub code, and multilingual datasets) with a focus on instruction following and tool‑calling. The training regime employed:
- Model size: 24 B parameters, 64 transformer layers.
- Tokenization: Mistral‑specific tokenizer with 32 k vocabulary.
- Compute: Approximately 1,200 GPU‑days on NVIDIA H100 GPUs (mixed‑precision FP16).
- Fine‑tuning: Instruction‑tuned on ~500 K high‑quality prompts, including function‑calling examples.
The FP8 quantized variant was produced post‑training using llm‑compressor (v0.6.1.dev0). This step does not alter the model’s weights beyond quantization, preserving the original knowledge while enabling faster, lower‑memory inference.
Licensing Information
The model card lists the license as unknown, but the tags include license:apache-2.0. In practice, this means the underlying base model (Mistral‑Small‑3.2‑24B‑Instruct‑2506) is released under the Apache 2.0 license, which is permissive and allows commercial use, modification, and distribution provided that:
- Original copyright notices and license text are retained.
- Any modifications are clearly marked.
- Patents, by the licensor are granted to downstream users.
If the FP8‑quantized derivative inherits the same license (the typical case when the quantization is performed by a third party without adding proprietary code), you may safely integrate it into commercial products, SaaS offerings, or research pipelines. Always double‑check the Hugging Face model card for any updated licensing statements before deployment.