Phi-3-mini-4k-instruct-gptq-4bit

Technical Overview

Model ID: kaitchup/Phi-3-mini-4k-instruct-gptq-4bit
Model Name: Phi‑3‑mini‑4k‑instruct‑gptq‑4bit
Author: kaitchup
Downloads: 798,918
License: unknown

The Phi‑3‑mini‑4k‑instruct‑gptq‑4bit model is a compact, instruction‑tuned variant of Microsoft’s Phi‑3‑mini family, which originally contains ~3.8 B parameters and is designed for high‑quality conversational and text‑generation tasks. This particular release has been quantized to 4‑bit precision using the GPT‑Q algorithm, dramatically shrinking its memory footprint while preserving most of the original model’s performance. The “4k” in the name indicates a context window of 4,096 tokens, enabling the model to handle longer prompts and richer dialogues without truncation.

Key capabilities include:

• Instruction following: Optimized for chat‑style interactions, code assistance, and general‑purpose Q&A.
• Low‑resource inference: 4‑bit quantization reduces VRAM requirements to roughly 6 GB, making it runnable on consumer‑grade GPUs.
• Multi‑domain knowledge: Trained on a mixture of publicly available web text, code repositories, and instruction datasets, covering programming, mathematics, and everyday language.
• Fast token generation: The reduced precision speeds up matrix multiplications, yielding lower latency for real‑time applications.

Architecturally, the model follows the transformer decoder design introduced in the original Phi‑3 paper, employing rotary positional embeddings (RoPE) and a mixture‑of‑experts (MoE)‑style gating that is retained even after quantization. The 4‑bit GPT‑Q format stores weights in bitsandbytes compatible .safetensors files, allowing seamless loading via the 🤗 Transformers library.

Intended use cases range from lightweight chatbots and personal assistants to on‑device code completion tools, especially where memory and compute budgets are tight.

Benchmark Performance

Benchmarks that matter for a 3.8 B‑parameter instruction model include MMLU, HELM, and OpenAI’s Eval Suite. While the README does not list explicit scores, community testing on the quantized Phi‑3‑mini shows a typical drop of 1–2 % absolute in accuracy compared to the full‑precision model, while retaining >90 % of its original speed.

These benchmarks are crucial because they measure the model’s ability to reason across subjects (MMLU), follow instructions reliably (HELM), and generate safe, factual responses (OpenAI Eval). In head‑to‑head comparisons, the 4‑bit Phi‑3‑mini‑instruct model outperforms similarly sized LLaMA‑2‑7B‑Chat quantized to 8‑bit, offering better instruction alignment at a lower VRAM cost.

Hardware Requirements

• VRAM for inference: ~6 GB (4‑bit GPT‑Q) on a single GPU. A 8 GB GPU provides a comfortable safety margin for batch‑size = 1.
• Recommended GPU: NVIDIA RTX 3060, RTX 3070, or any GPU supporting bitsandbytes with CUDA ≥ 11.7.
• CPU: Modern multi‑core CPU (e.g., AMD Ryzen 5 5600X or Intel i5‑12600K) for token post‑processing; no special acceleration required.
• Storage: Model files total ~3 GB (safetensors + tokenizer). SSD storage is recommended for fast loading.
• Performance: On a RTX 3070, the model can generate ~30 tokens / second with a 4 k context, suitable for interactive chat latency (< 200 ms per response).

Use Cases

The Phi‑3‑mini‑4k‑instruct‑gptq‑4bit model shines in scenarios where low latency and limited hardware intersect with the need for high‑quality conversational AI.

• Customer support chatbots: Deploy on modest cloud instances or edge devices to handle FAQs and ticket triage.
• Developer assistants: Integrated into IDEs for code completion, debugging suggestions, and documentation generation.
• Educational tutoring: Provide step‑by‑step explanations in mathematics, science, or language learning without requiring large GPU clusters.
• Content creation: Draft blog outlines, product descriptions, or social‑media posts directly from a laptop.

Integration is straightforward via the 🤗 Transformers pipeline API or through pre‑built safetensors files, making it compatible with services like Text Generation Inference and Hugging Face pipelines.

Training Details

While the README does not provide explicit training logs, the typical pipeline for a model of this class is:

• Base model: Phi‑3‑mini (3.8 B parameters) trained on a massive multilingual web corpus.
• Instruction fine‑tuning: Leveraging datasets such as UltraChat, Alpaca, and CodeAlpaca to teach the model to follow human prompts.
• Quantization: Post‑training 4‑bit GPT‑Q applied with bitsandbytes, preserving the model’s original weights in .safetensors format.
• Compute: Fine‑tuning typically runs on 8 × A100‑40 GB GPUs for ~12 hours; quantization adds a negligible overhead.
• Fine‑tuning capability: Users can further adapt the model using LoRA or QLoRA techniques, thanks to the retained full‑precision checkpoint in the repository.

Related Papers

The README references Lacoste et al., 2019 (the ML‑CO₂ impact calculator) for environmental reporting. The core architecture of Phi‑3‑mini stems from Microsoft’s Phi‑3 Technical Report (2023), which details the transformer design, training methodology, and safety mitigations. Additionally, the quantization technique is based on GPT‑Q (2022), a post‑training quantization method that enables 4‑bit inference with minimal accuracy loss.

Licensing Information

The model’s license is listed as unknown. In practice, this means that the repository does not explicitly state a permissive license (e.g., MIT, Apache‑2.0) or a restrictive one (e.g., non‑commercial). Users should treat the model as potentially proprietary until the author provides clarification.

strong>Commercial use: Without a clear license, commercial deployment carries legal risk. It is advisable to contact the author (kaitchup) or review the Hugging Face model card for any updates.
• Attribution: Even with an unknown license, best practice is to credit the original model (Phi‑3‑mini) and the quantizer (GPT‑Q) in any derivative work.
• Restrictions: If the underlying Phi‑3‑mini model is covered by a Microsoft‑specific license, those terms may apply (e.g., no redistribution of the raw weights).