Best Quantization Formats in 2026: GGUF, AWQ, GPTQ, EXL2

Quick Answer

As of September 2026, the top pick is GGUF (GGML Universal File) Q4_K_M with llama.cpp (Georgi Gerganov and contributors, llama.cpp ecosystem) [17][5]. In order, the ranking is GGUF (GGML Universal File) Q4_K_M with llama.cpp (Georgi Gerganov and contributors, llama.cpp ecosystem), EXL2 4.0–4.5 bpw with ExLlamaV2 (turboderp), Activation-aware Weight Quantization (AWQ) 4-bit (MIT Han Lab, Hugging Face Transformers and vLLM implementations), Generative Pre-trained Transformer Quantization (GPTQ) 4-bit (IST Austria and ETH Zürich; AutoGPTQ, Transformers, vLLM, Marlin implementations), GGUF (GGML Universal File) Q5_K_M (Georgi Gerganov and contributors, llama.cpp ecosystem), and GGUF (GGML Universal File) Q8_0 (Georgi Gerganov and contributors, llama.cpp ecosystem) [17][5].

Key Takeaways

• GGUF (GGML Universal File) Q4_K_M with llama.cpp (Georgi Gerganov and contributors, llama.cpp ecosystem) is the broadest local inference option for CPU, Apple Silicon, and GPU offload [17].
• EXL2 4.0–4.5 bpw with ExLlamaV2 (turboderp) was the fastest Qwen3-8B NVIDIA option in a 2026 test, at 205.4 tokens/s and 4.6 GB [12].
• Activation-aware Weight Quantization (AWQ) 4-bit from MIT Han Lab is the common GPU-serving choice in vLLM stacks [16].
• Generative Pre-trained Transformer Quantization (GPTQ) 4-bit with IST Austria and ETH Zürich implementations was measured at 178.2 tokens/s on Qwen3-8B NVIDIA hardware [12].
• GGUF (GGML Universal File) Q5_K_M is the safer quality/size compromise: one 2026 benchmark reported 5.5 bits/weight, 5.27 perplexity, and about +1.2% over its FP16 reference [5].
• GGUF (GGML Universal File) Q8_0 is for near-FP16 quality when memory allows: one 2026 benchmark reported 8.5 bits/weight, 5.22 perplexity, and about +0.2% relative degradation [6].

How do GGUF, AWQ, GPTQ, and EXL2 compare on quality, speed, and VRAM?

Which quantization format should I use for local LLM inference?

1. GGUF (GGML Universal File) Q4_K_M with llama.cpp

GGUF (GGML Universal File) Q4_K_M with llama.cpp is maintained in the llama.cpp ecosystem by Georgi Gerganov and contributors. The format ranks here because a single-file checkpoint runs on CPU, Apple Silicon, and GPU-offloaded local stacks such as llama.cpp, Ollama, and LM Studio [17][16][23][31]. A Llama 2 13B comparison measured Q4_K_M at 4.5-bit nominal precision, 7.502 GB file size, 8,985 MB VRAM, 4.33326 perplexity, and 30.83 tokens/s [1][5]. A Qwen3-8B NVIDIA test measured 165.1 generation tokens/s with an approximately 5.0 GB file [12].

The practical use is general local inference where portability and memory fit matter more than peak NVIDIA serving speed [16][23]. Q4_K_M suits ordinary chat, summarization, retrieval-augmented generation, and everyday coding in llama.cpp-compatible stacks when hardware can hold the checkpoint [17][12]. The same Qwen3-8B comparison found EXL2 at 205.4 tokens/s, GPTQ-Marlin at 178.2 tokens/s, and AWQ at 155.3 tokens/s, so GGUF is not the fastest in that setup [12].

The caveat is that these figures are test-specific, not a universal quality score. Perplexity, VRAM, tokens/s, and perceived quality vary with model revision, context length, calibration data, runtime kernels, and GPU memory headroom [10][12]. Choose Q4_K_M for broad local deployment, then validate on your target task before assuming it meets a stricter accuracy bar.

2. EXL2 4.0–4.5 bpw with ExLlamaV2

EXL2 4.0–4.5 bpw with ExLlamaV2 [16][10], maintained by turboderp [16], ranks here because the cited Qwen3-8B NVIDIA test measured 205.4 tokens/s at 4.0 bits per weight from a 4.6 GB file, about 24% faster than GGUF Q4_K_M in that setup [12], while a separate 2026 comparison estimated EXL2 4.5-bit quality loss at +0.08–0.15 perplexity versus FP16 [10].

EXL2 runs on NVIDIA CUDA GPUs through ExLlamaV2 or ExLlamaV3, with the latter reportedly requiring CUDA 12.4 or newer [17]. The format supports variable average bitrates, approximately 2–8 bits per weight, by mixing quantization levels across layers [16]. A 70B model at 4.0 bits per weight is estimated to need roughly 34–38 GB of VRAM before context and runtime overhead [27], so local deployment usually requires a high-memory NVIDIA card rather than a CPU-only machine [16][17].

Use EXL2 when the goal is maximum single-user NVIDIA throughput, minimum model size [12], or exact VRAM tuning in ExLlamaV2; it was the fastest format in the cited Qwen3-8B test [12]. The caveat is that speed depends on calibration data, group size, context length, hardware, and runtime, so the figure should not be treated as a universal leaderboard ranking [10][16].

3. Activation-aware Weight Quantization (AWQ) 4-bit

Activation-aware Weight Quantization (AWQ) 4-bit [14], from MIT Han Lab, with Hugging Face Transformers and vLLM implementations, ranks here because it packs a compact weight footprint into a GPU-serving stack that many local NVIDIA deployments already use [16]. The format commonly targets CUDA GPUs through runtimes such as vLLM or SGLang, making it a practical choice when server-side throughput matters more than CPU portability [17][25].

Key specs from public 2026 tests show AWQ 4-bit at about 4.0 bits per weight, with one Llama 2 13B comparison reporting 4.32522 perplexity, 7.624 GB of file size, 10,567 MB VRAM, and 39.47 tokens/s [1]. A Qwen3-8B NVIDIA test measured 155.3 generation tokens/s with an approximately 5.2 GB model file, while a separate quality benchmark reported roughly +1.7% relative degradation versus its reference [5][12].

Hardware needs are primarily a modern NVIDIA CUDA GPU, enough VRAM for the quantized weights plus context and KV-cache overhead, and a compatible serving runtime such as vLLM [17][18]. AWQ suits API-style local inference, batch generation, or production-like deployments where the stack already favors AWQ kernels. A caveat: reported perplexity and speed values are model- and hardware-specific, so task-specific evaluation remains necessary before adopting AWQ as a universal replacement for GGUF Q4_K_M [5][10][12].

4. Generative Pre-trained Transformer Quantization (GPTQ) 4-bit

Generative Pre-trained Transformer Quantization (GPTQ) 4-bit comes from IST Austria and ETH Zürich; AutoGPTQ, Transformers, vLLM, Marlin implementations make it a practical GPU-serving format [14][16][25]. It ranks here because the Hugging Face Transformers documentation identifies a GPTQ benchmark model, while a Qwen3-8B NVIDIA comparison measured 178.2 tokens/s for GPTQ-Marlin at about 5.1 GB [14][12]. A separate 2026 comparison estimated 0.08–0.18 perplexity above FP16 and a 2–5% coding Pass@1 impact for a 7B-class model [10].

A tested Qwen3-8B GPTQ-Marlin configuration used 4-bit weights with 128-group quantization in a vLLM-style stack [12]. A Llama 3.3 70B-class GPTQ 4-bit footprint is reported around 38 GB of VRAM, before context and KV-cache overhead [18]. An RTX 4090 test measured 94 generation tokens/s and about 3,900 prompt-processing tokens/s for GPTQ [4]. The format needs an NVIDIA CUDA GPU with enough memory for the quantized weights plus cache; some runtimes mention AMD ROCm support, so check the target stack [17][18][29][31].

Use GPTQ when serving through vLLM, TGI, SGLang, or AutoGPTQ and compatibility with older GPU workflows matters [16][17][25]. A caveat is that an A100 test for a Mistral 7B model showed 8.78 seconds for GPTQ versus 4.96 seconds for AWQ, so benchmark the exact model and server before selecting it [22].

5. GGUF (GGML Universal File) Q5_K_M

GGUF (GGML Universal File) Q5_K_M from Georgi Gerganov and contributors, llama.cpp ecosystem, appears here because the reported quality loss is smaller than the 4-bit options while adding only modest memory.[5][10] A 2026 benchmark measured 5.5 bits/weight, 5.27 perplexity, and about +1.2% relative to its FP16 reference.[5] Another comparison estimated +0.05–0.10 perplexity versus FP16 and a 1–3% coding Pass@1 impact for a 7B-class model, at approximately 5.5 GB VRAM.[10]

The GGUF runtime supports CPU inference, Apple Silicon, and GPU offload through llama.cpp, Ollama, or LM Studio.[16][17] Q5_K_M suits workloads where the reported +0.05–0.10 perplexity[10] and 1–3% coding Pass@1 impact[10] are acceptable, while a slightly larger memory footprint than a 4-bit build[5][10] is available. In practice, allocate at least the reported approximately 5.5 GB VRAM[10] plus context KV cache before choosing hardware.

A caveat is that these figures come from separate evaluations with different models, datasets, calibration procedures, and baselines.[5][10][12] Q5_K_M should therefore be validated on the target model and task before treating the reported loss as universal.

6. GGUF (GGML Universal File) Q8_0

GGUF (GGML Universal File) Q8_0 [6] from Georgi Gerganov and contributors, llama.cpp ecosystem is an 8-bit zero-point GGUF variant [6] that ranks here for quality-sensitive local work when memory allows. The variant reports 8.5 bits/weight, 5.22 perplexity, and approximately +0.2% relative degradation [6]. That profile suits evaluation reproducibility, subtle instruction-following, structured output, or numerically sensitive tasks when the model fits available memory [6].

Hardware support comes from llama.cpp-compatible GGUF runtimes that cover CPU inference, Apple Silicon, and GPU offload [17][16]. Use Q8_0 on a local machine when FP16 weights do not fit but near-FP16 quality is still needed [6]. An RTX comparison reports Q8_0 at approximately 68 generation tokens/s on an NVIDIA RTX 4090 and 32 tokens/s on an NVIDIA RTX 3060, below GGUF Q4_K_M at approximately 105 tokens/s [4].

The practical caveat is that the higher weight precision reduces generation throughput in that RTX comparison and leaves less memory headroom for context [4][6].

What hardware supports GGUF, AWQ, GPTQ, and EXL2?

GGUF (GGML Universal File), used with llama.cpp [17][16], supports the broadest hardware range [17][16]: CPUs, Apple Silicon, and GPU offload, while Activation-aware Weight Quantization (AWQ), GPTQ (Generative Pre-trained Transformer Quantization), and EXL2 are primarily NVIDIA CUDA GPU formats [17][16]. For a laptop, desktop, or server without a discrete graphics processor, GGUF remains the practical default because llama.cpp, Ollama, and LM Studio can run quantized models on mixed CPU/GPU systems [17][16][25].

AWQ is best suited to NVIDIA CUDA GPUs in production serving stacks such as vLLM [17][16][31]. The format is commonly described as GPU-first for modern NVIDIA cards [21][34]. Its 4-bit kernels aim to reduce memory use while keeping inference on the accelerator [21].

GPTQ also targets NVIDIA CUDA GPUs and remains supported in major inference runtimes [17][29][31]. AMD ROCm support appears in certain GPTQ-compatible runtimes, making it less portable than GGUF but still usable beyond a single GPU vendor where the stack permits [17][33].

EXL2, the variable-bit-per-weight format for ExLlamaV2 [16][21], is effectively NVIDIA-only in mainstream use because ExLlamaV2 and related runtimes depend on CUDA [17][16]. ExLlamaV3 requires CUDA 12.4 or newer, so the hardware choice must include a supported NVIDIA GPU and current driver stack [21].

The practical split is: choose GGUF for CPUs, Apple Silicon, and mixed devices; choose AWQ or GPTQ for NVIDIA serving; choose EXL2 when tuning ExLlamaV2 throughput on a single NVIDIA GPU [17][16][23].

How much VRAM does a quantized local model need?

A quantized local model typically needs enough VRAM to hold its quantized weights plus context-dependent KV-cache and runtime overhead; for a 70B-class model at roughly 4-bit precision, published estimates cluster around 36–40 GB [18].

For a Llama 3.3 70B-class checkpoint, the comparison reports GGUF (GGML Universal File in the llama.cpp ecosystem) Q4_K_M at approximately 40 GB of VRAM, GPTQ (Generative Pre-trained Transformer Quantization documented by Hugging Face Transformers) 4-bit at approximately 38 GB, AWQ (Activation-aware Weight Quantization documented by Hugging Face Transformers) 4-bit at approximately 36 GB, and EXL2 (ExLlamaV2 variable-bit-per-weight format maintained by turboderp) at 4.5 bits/weight at approximately 40 GB [14][18][25][27].

A separate estimate places a 70B model in EXL2 at 4.0 bits/weight at approximately 34–38 GB of VRAM, depending on context and runtime overhead [27].

For a Llama 2 13B model, one test measured GGUF Q4_K_M at 8,985 MB of VRAM and AWQ 4-bit at 10,567 MB in the same comparison [1].

A general sizing estimate gives approximately 8 GB minimum VRAM for a 13B model at INT4, without full context/KV-cache specification [24].

Which runtime works with GGUF, AWQ, GPTQ, and EXL2?

GGUF (GGML Universal File) runs through llama.cpp-compatible runtimes such as llama.cpp, Ollama, and LM Studio; AWQ (Activation-aware Weight Quantization) and GPTQ (Generative Pre-trained Transformer Quantization) run primarily through GPU-serving stacks such as vLLM; EXL2 runs through turboderp's ExLlamaV2 or ExLlamaV3 on NVIDIA CUDA hardware [17][16][12][1].

The llama.cpp ecosystem, maintained by Georgi Gerganov and contributors, supports CPU, Apple Silicon, and GPU offload for GGUF, which makes it the broadest-hardware option among the four formats [1][17]. Ollama and LM Studio also load GGUF files directly in common local workflows [16].

AWQ is commonly used with vLLM, SGLang, Hugging Face Transformers, or TensorRT-LLM for NVIDIA CUDA GPUs; GPTQ remains supported in major inference stacks, although many 2026 guides describe it as increasingly superseded by AWQ [17][16][21]. The Hugging Face Transformers documentation identifies benchmark models for both AWQ and GPTQ quantized inference [14].

EXL2 uses variable average bitrates of about 2–8 bits per weight in ExLlamaV2, so a local NVIDIA user can tune the model to a fixed VRAM budget [16]. ExLlamaV3 requires CUDA 12.4 or later [17].

Choose GGUF when CPU, Apple Silicon, Ollama, LM Studio, llama.cpp compatibility, or flexible offloading matters; choose AWQ or GPTQ for a vLLM-style GPU-serving stack; choose EXL2 when NVIDIA local throughput and exact VRAM tuning matter more than portability [17][16][23].

Frequently Asked Questions

Which quantization format suits CPU and Apple Silicon local inference?

GGUF, the GGML Universal File format maintained in the llama.cpp ecosystem by Georgi Gerganov and contributors, is the broadest-hardware choice because it runs on central processors, Apple Silicon, and GPU offload with tools such as Ollama and LM Studio [17][25]. Choose Q4_K_M for ordinary local chat when memory matters; a 13B-model test measured 7.502 GB and 8,985 MB VRAM [1].

Which format gave high single-user NVIDIA GPU generation throughput?

EXL2, a variable-bit-per-weight quantization format for ExLlamaV2 maintained by turboderp, measured 205.4 tokens/s at 4.0 bpw on an 8B model with a 4.6 GB file in an NVIDIA test [12]. GPTQ-Marlin / vLLM measured 178.2 tokens/s and GGUF Q4_K_M measured 165.1 tokens/s in that comparison [12].

How do AWQ and GPTQ compare for vLLM GPU serving?

AWQ, Activation-aware Weight Quantization associated with MIT Han Lab and implemented in Transformers/vLLM ecosystems, is a common NVIDIA CUDA GPU-serving choice [16][17]. In an 8B-model NVIDIA comparison, AWQ / vLLM produced 155.3 tokens/s while GPTQ-Marlin, a Generative Pre-trained Transformer Quantization method introduced at IST Austria and ETH Zürich, produced 178.2 tokens/s [12]. A 7B-model test on an A100-80GB GPU gave 4.96 seconds for AWQ versus 8.78 seconds for GPTQ [22].

How much VRAM should I budget for a large dense model in low-bit formats?

For a 70B-class dense model, estimates are approximately 36 GB for AWQ 4-bit, 38 GB for GPTQ 4-bit, 40 GB for EXL2 at 4.5 bits/weight, and 40 GB for GGUF Q4_K_M [18]. A separate estimate places a 70B model in EXL2 at about 34-38 GB when tuned to 4.0 bits per weight [27].

Does the quantization format change the model license?

No. GGUF, AWQ, GPTQ, and EXL2 do not impose one universal model license; the quantized checkpoint inherits the underlying base-model license, plus any runtime or converter terms you use [16][17][25]. Redistribution rights depend on the selected model repository and project terms, so check the model card separately before sharing a quantized file publicly [17][22].

How do I set up GGUF, AWQ, GPTQ, and EXL2 locally?

GGUF is usually the simplest setup because a single file can run in llama.cpp, Ollama, or LM Studio without building a separate quantization pipeline [17][23][25]. AWQ and GPTQ generally need GPU-serving stacks such as vLLM, SGLang, or Transformers, plus calibration during quantization; EXL2 needs ExLlamaV2/ExLlamaV3 or TabbyAPI on CUDA hardware [16][17].

Sources

1. A detailed comparison between GPTQ, AWQ, EXL2, q4_K_M ... — accessed 2026-09-24
2. GGUF vs GPTQ vs AWQ vs EXL2: Model Quantization Format ... — 2026-04-08
3. Reasoning Source files for GGUF, EXL2, AWQ, GPTQ - Hugging Face — 2025-03-04
4. Quantized Local LLMs: 4-bit vs 8-bit Performance Analysis — 2026-03-05
5. Local LLM Quantization Quality Benchmarks 2026 - Presenc AI — 2026-05-07
6. LLM Quantization: GGUF, AWQ, GPTQ, and FP8 Comparison with ... — 2025-08-21
7. DavidAU/Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709 ... — 2026-09-17
8. AWQ vs GGUF vs GPTQ: Quantization Methods Compared for AI 2026 — accessed 2026-09-24
9. Quantization Methods Comparison | oobabooga/oobabooga.github ... — 2025-12-08
10. GGUF vs GPTQ vs EXL2: LLM Quantization Compared [2026] — 2026-08-16
11. GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained ... — 2026-09-18
12. GGUF vs EXL2 vs AWQ: Which Is Fastest on NVIDIA in 2026? — 2026-05-03
13. DavidAU/Qwen3.5-9B-Claude-4.6-OS-Auto-Variable-HERETIC ... — 2026-08-03
14. Quantize 🤗 Transformers models — accessed 2026-09-24
15. GPTQ vs AWQ vs GGUF: Speed, Accuracy & Cost Compared (2026) — 2026-09-11
16. GGUF vs AWQ vs GPTQ vs MLX: LLM Quant Formats 2026 — 2026-06-28
17. GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026) — 2026-09-18
18. Quantization LLM : Comparaison GPTQ, AWQ, EXL2, GGUF — 2026-05-16
19. GGUF vs EXL2 vs AWQ vs GPTQ - Local AI Zone — 2026-08-15
20. Quantization Formats Explained - GGUF, GPTQ, AWQ, EXL2 - Cloudzy — 2026-07-02
21. Model Formats Explained: GGUF vs GPTQ vs AWQ vs EXL2 — 2026-01-30
22. LLM Quantization Methods: GPTQ, AWQ, GGUF — 2026-09-04
23. GGUF vs GPTQ vs AWQ: Quantization Methods Compared — 2026-06-12
24. Best NVIDIA GPUs for LLMs in 2026: Ranked by Use Case — 2026-09-08
25. GGUF, safetensors, MLX, GPTQ, AWQ, EXL2 - Inventive HQ — 2026-06-25
26. VRAM, Shared Memory, and How Much You Actually Need — 2026-07-28
27. Deploy Aphrodite Engine on GPU Cloud: EXL2, GGUF ... — 2026-07-04
28. A detailed comparison between GPTQ, AWQ, EXL2, q4_K_M, q4_K_S, and load_in_4bit: perplexity, VRAM, speed, model size, and loading time. — 2023-10-25
29. Understanding LLM Quantization: GGUF, GPTQ, AWQ, EXL2 ... — 2026-04-08
30. GGUF vs GPTQ vs AWQ — LLM Quantization Formats Explained (2026) — 2026-05-09
31. Quantization Format Comparison 2026 - Presenc AI — 2026-05-23
32. Resource-Efficient Language Models:Quantization for Fast ... - arXiv — accessed 2026-09-24
33. Quantization for Local LLMs: How It Works and Which Formats ... — 2025-09-05
34. LLM Quantization Formats Compared: GGUF vs MLX vs EXL3 vs ... — 2026-07-20
35. Machine Learning Category - Page 534 of 591 — 2026-09-18
36. AI Infrastructure Category - Page 26 of 28 — 2026-09-19
37. For Devs Category - MarkTechPost — 2026-09-19
38. Comparison Category — 2026-09-18
39. Bartowski turns GGUF quantization heuristics into per-tensor maps — 2026-09-11
40. Archive - The Kaitchup – AI on a Budget — 2026-09-19
41. BitNet - 1-bit LLM Inference on CPU | AI/TLDR — 2026-09-10
42. RealRebelAI Shrinks Alibaba's Qwen-Image 2.1 to Fit ... — 2026-09-20
43. SemIf (formerly OpenJev) — typed decisions… - AI/TLDR — 2026-09-18
44. Archsloth fixes two AutoRound flags, says Qwen3-4B drifts less — 2026-09-14
45. AI Engineering Insider (@aiengineeringinsider) - Substack — 2026-08-30

Related on Q4KM

• AI Model Quantization Guide: Techniques, Trade-offs, and Performance