Best Open-Source LLMs to Run Locally in 2026

Rankings 2026-09-25 Last updated 2026-09-25 14 min read By Q4KM

Quick Answer

The top pick is Qwen3-32B (Alibaba Qwen), as of September 2026, because it offers Apache 2.0 licensing and workstation-level feasibility for local inference [1][3].
The ranking is Qwen3-32B (Alibaba Qwen), Gemma 4 31B IT (Google), Devstral 2 (Mistral AI), GLM-5.3-Flash (Z.ai), DeepSeek-V4.1-Flash (DeepSeek), and DeepSeek-V4-Pro (DeepSeek) [1][3][5][8][9][11][14][15].

Key Takeaways

Which local LLMs compare well on specs and benchmarks?

Model Org Params Quant/VRAM Context License Key benchmark (date)
Qwen3-32B Alibaba / Qwen Team [1][3] 32B dense flagship checkpoint [1][3] ≈18–22 GB at 4-bit [1][3] Commonly 128K; checkpoint-dependent [1][3] Apache 2.0 [1][3] not published [1][3]
Gemma 4 31B IT Google [11][15] 31B dense candidate in 2B–31B family [11][15] ≈18–22 GB at 4-bit [11][15] Up to 256K tokens [11][15] Apache 2.0 [11][15] not published [11][15]
Devstral 2 Mistral AI [14] 123B [14] ≈65–75 GB at 4-bit [14] 256K tokens [14] Mistral license [14] not published [14]
GLM-5.3-Flash Z.ai [9] 320B total / 18B active [9] ≈160–180 GB at 4-bit [9] 1,048,576 tokens [9] MIT [9] not published [9]
DeepSeek-V4.1-Flash DeepSeek [8] 552B total; approximately 8B prefill and 16B decode configuration reported [8] ≈280–310 GB at 4-bit [8] 1M tokens [8] MIT [8] 90.6 Terminal-Bench 2.1, vendor-reported, June–September 2026 [8]
DeepSeek-V4-Pro DeepSeek [5] 1.6T total / 49B active [5] ≈800–900 GB at 4-bit [5] 1M tokens [5] MIT [5] not published [5]

What are good open-source LLMs to run locally?

1. Qwen3-32B

Qwen3-32B from Alibaba Qwen is one of the strongest commercially permissive local models in 2026, earning its #1 spot for Apache 2.0 licensing and workstation feasibility [1][3]. The public dense flagship checkpoint has 32 billion parameters, requires about 18–22 GB under a 4-bit weight estimate, and commonly supports 128K tokens of context, with exact limits depending on the selected checkpoint [1].

Hardware-wise, plan for a 24 GB GPU based on that weight estimate [1], such as an NVIDIA GeForce RTX 4090 with 24 GB GDDR6X and 1,008 GB/s bandwidth, priced at $1,599–$1,999 in April 2026 [20]. A 32 GB NVIDIA GeForce RTX 5090, priced at $1,999–$2,199 in April 2026, gives more headroom for context [20][3].

The model’s 4-bit weight estimate leaves room for KV cache only if context length and batch size stay modest; actual VRAM rises with runtime buffers, quantization metadata, and longer prompts [1][3]. Qwen3-32B is a strong choice for a general-purpose local assistant on a workstation GPU, but it is not a laptop-class model when long context is needed [1][3].

2. Gemma 4 31B IT

Gemma 4 31B IT, published in the Gemma 4 family by Google, ranks here as a permissive dense model for local use [11][15]. The Gemma 4 family includes checkpoints from 2B to 31B parameters, with gemma-4-31B-it as the largest dense candidate and an Apache 2.0 license [11][15]. At 4-bit quantization, the 31B checkpoint is estimated to occupy about 18–22 GB for weights, plus overhead for scales, KV cache, and runtime buffers; context can reach up to 256K tokens [11][15].

For a single-GPU workstation, target at least 24 GB VRAM for 4-bit inference, based on the 31B checkpoint's 18–22 GB weight-only estimate and local-model guidance for 24 GB tiers [1][11][15]. KV cache grows with sequence length, so long-context use needs extra headroom. Gemma 4 31B IT suits private general assistants, summarization, structured extraction, and local coding workflows where Apache 2.0 licensing matters [11][15]. The caveat is that the memory estimate excludes full runtime overhead; using a large fraction of the 256K-token window can exceed one consumer GPU [1][11][15].

3. Devstral 2

Devstral 2 by Mistral AI is the coding-focused entry in this local list [14]. The September 2026 catalog identifies it as a software-engineering model, giving it a clear role for self-hosted developer workflows on hardware that can hold large quantized weights and long context windows without relying on hosted API access [14].

The reported specs are 123B parameters, a 256K-token context window, and an approximate 65–75 GB memory footprint at 4-bit precision [14]. The memory figure is an engineering estimate, not an official specification, so KV cache, runtime buffers, context length, and batch size will increase VRAM use [14].

Hardware should be a multi-GPU workstation or server-class GPU setup, because the estimated weight footprint exceeds common 32 GB consumer cards [14][20]. The best local use is software-engineering assistance for production coding teams where model context and codebase access matter [14]. One caveat is that Devstral 2 uses a Mistral license rather than Apache 2.0 or MIT, so commercial terms need review before production deployment [14].

4. GLM-5.3-Flash

GLM-5.3-Flash from Z.ai ranks here as a large local option, listed with 320B total and 18B active parameters [9]. The model carries an MIT license, supports 1,048,576 tokens of context, and is estimated at about 160–180 GB for a 4-bit build [9]. Its reported API pricing is $0.15 per million input tokens and $0.50 per million output tokens, as of September 2026 [9].

Hardware planning should start from the estimated 160–180 GB weight footprint, then add KV-cache, quantization metadata, and runtime buffer overhead [9]. A practical deployment therefore needs a multi-GPU server with hundreds of gigabytes of combined GPU memory, not a laptop or ordinary desktop workstation [9]. Use GLM-5.3-Flash for long-context local workflows where data control matters and the team already owns that class of inference hardware [9].

The main caveat is memory: a 1,048,576-token context can increase KV-cache demand beyond the weight estimate, depending on batch size and serving engine [9]. Verify actual VRAM usage with your chosen quantization and runtime before committing to the model for production traffic [9].

5. DeepSeek-V4.1-Flash

DeepSeek-V4.1-Flash, from DeepSeek, ranks here because the model combines MIT licensing, a 1M-token context window, public weights, and strong coding and agentic positioning in September 2026.[8]

The architecture has 552B total parameters, with a reported configuration using approximately 8B prefill and 16B decode parameters.[8] At roughly 4-bit precision, the weights need about 280–310 GB of memory, before KV-cache growth from long context increases the required VRAM.[8] The 1M-token context window means the KV cache can add substantial memory demand as prompts lengthen.[8]

Best use is local coding and agentic workflows on a multi-GPU server, because the estimated 4-bit footprint leaves no room for desktop VRAM.[8] A practical server setup should use multiple high-capacity GPUs with enough available aggregate VRAM to hold the 4-bit weights and KV cache.[8] The caveat is that the 90.6 Terminal-Bench 2.1 score is vendor-reported for June–September 2026, so treat it as dated rather than independently comparable.[8]

6. DeepSeek-V4-Pro

DeepSeek-V4-Pro is a DeepSeek model for multi-GPU local deployment: it has 1.6T total parameters, 49B active parameters, MIT licensing, and a 1M-token context window.[5] The release date is April 24, 2026, placing the model among the newer large open-weight candidates in the 2026 list.[5] Its mixture-of-experts design activates only part of the weights per token, which keeps inference compute lower while still requiring storage for most experts locally.[5][24]

Hardware needs are server-class, not laptop-class: a single NVIDIA GeForce RTX 5090 with 32 GB GDDR7 is too small for the 1.6T-parameter model.[20][5] Multi-GPU systems using enterprise cards such as NVIDIA B200 with 192 GB HBM3e and NVIDIA H100 SXM with 80 GB HBM3 are better matches.[23] The practical setup is an enterprise local-inference stack using vLLM, GPU servers, and containers.[23]

The best use is high-capacity local serving for enterprise workloads that need a very large MoE model on their own hardware.[23][30] One caveat is memory overhead: the 1M-token context window can make KV-cache and runtime buffers consume substantial additional GPU memory, so total VRAM needs exceed weight-only estimates.[5][23]

How much VRAM should you buy for local LLM inference?

Buy at least 16 GB of VRAM for useful quantized local LLM inference, choose 24–32 GB for stronger mid-sized models, and buy 80 GB or more only for multi-GPU server-class work [1][3][20]. An 8 GB system can run small Alibaba Qwen checkpoints for basic chat, but it leaves little room for long context [1].

A 16 GB card fits a quantized Google Gemma 4 checkpoint and supports mid-sized local assistants [1][3]. A 24 GB card can run stronger coding-oriented checkpoints such as Alibaba's Qwen3-Coder-30B on a single GPU, which made it the best all-round local coding model in one June 2026 test [1][33].

A 32 GB NVIDIA GeForce RTX 5090 gives more headroom for long context, with reported throughput of 26–34 tok/s on 70B Q4 workloads and over 95 tok/s on 8B Q4 workloads [20].

For larger local deployments, DeepSeek's 685B-parameter DeepSeek V3.2 requires about 342.5 GB of weights-only storage at 4-bit precision, so plan on multiple 80 GB HBM cards such as NVIDIA H100 SXM in server configurations [24][23]. The price gap is large: an RTX 5090 costs $1,999–$2,199, while one H100 SXM can cost roughly $25,000–$35,000 [20][23].

Which licenses allow commercial use of local LLM weights?

Apache License 2.0 and MIT licenses allow commercial use of local large language model weights [1][3][17].
For example, Qwen3 from Alibaba and the Qwen Team is listed under Apache License 2.0 [1][3], Gemma 4 from Google includes Apache 2.0 checkpoints such as gemma-4-31B-it [11][15], DeepSeek-V4.1-Flash from DeepSeek uses MIT licensing [8], and GLM-5.3-Flash from Z.ai is reported under MIT terms [9].
Open-weight release is not the same as open source, because public weights can carry custom terms such as Kimi K3 license from Moonshot AI or GLM-5.2 license from Z.ai [8][13]. A downloadable repository does not settle commercial permissions by itself; engineers should read the model card and license text before redistribution, internal deployment, or product integration [17][27]. The distinction matters when a model card is separated from the downloadable checkpoint [4][17].
The cited comparisons use Apache License 2.0 [1] and MIT [8] as practical permissive labels for local inference workloads, while custom licenses require separate, formal legal review before release [3][9][13]. Keep the exact license file with each downloaded checkpoint [27].

What is the difference between open-weight and open-source models?

Open-weight means published model weights are downloadable for local use, while open-source adds permissive licensing that covers reuse rights.[37]

Public checkpoints can also still carry custom terms. Qwen3 from Alibaba / Qwen Team is listed under Apache 2.0[1][3]. Gemma 4 from Google is cataloged under Apache 2.0[11][15]. DeepSeek-V4.1-Flash from DeepSeek carries MIT licensing[8].

Kimi K3 from Moonshot AI uses a custom Kimi K3 license[8]. GLM-5.2 from Z.ai uses the GLM-5.2 license[13]. For commercial use, teams should check each repository's file before assuming broader reuse rights[27][17].

Open-weight status does not guarantee the same reuse rights as an open-source license[27][17]. A public checkpoint can allow local inference while limiting later commercial distribution or actual product use[27][17].

The practical difference is legal permission: open-weight gives access to weights; open-source gives a broader permission set.[37] A model can run locally under either category, but only the license defines what developers may ship, modify, and sell.[37]

Frequently Asked Questions

How much memory should I budget for a local LLM in 2026?

Current guidance maps 8–12 GB RAM to small models, 16 GB to quantized mid-sized checkpoints, 24 GB VRAM to Qwen3-Coder 30B from the Qwen Team at Alibaba or Gemma 4 26B A4B from Google, and 32–64 GB unified memory to larger distills [1]. A 27B model estimated at about 13.5 GB of 4-bit weights usually needs more than a 16-GB card for long context because the KV cache adds overhead [8].

Is open source the same as open weights?

No; public weights can carry custom or restricted terms, so check each license before commercial use [17][27]. The candidate set includes Apache 2.0 models such as Qwen3-32B from the Qwen Team at Alibaba [1] and Gemma 4 31B IT from Google [11], while DeepSeek-V4.1-Flash from DeepSeek is reported under MIT [8].

Which local model is good for coding on a single GPU?

For single-GPU coding, a June 2026 guide reports Qwen3-Coder 30B from the Qwen Team at Alibaba fits one 24 GB GPU and was the best all-round local coding model in that test [33]. A related dense option, Qwen3.6-27B from the Qwen Team at Alibaba, is reported at 77.2% on SWE-bench Verified as of April 22, 2026 [8].

Do DeepSeek models need a server to run locally?

Yes, for large DeepSeek releases local inference is a multi-GPU/server workload rather than an ordinary desktop task [5][24]. DeepSeek-V4-Pro from DeepSeek has 1.6T total and 49B active parameters with an engineering estimate of about 800–900 GB at 4-bit, while DeepSeek V3.2’s 685B total parameters require hundreds of gigabytes even for weights-only storage [5][24].

Which consumer GPU is the practical choice for local LLMs in 2026?

NVIDIA RTX 5090, priced at $1,999–$2,199 in April 2026 street reports, has 32 GB GDDR7 and 1,792 GB/s bandwidth, with reported throughput of 26–34 tok/s for 70B Q4 models [20]. NVIDIA RTX PRO 6000 Blackwell offers 96 GB GDDR7 and was reported at 8,425 tok/s on Qwen3-Coder-30B from the Qwen Team at Alibaba through vLLM [18].

What is the easiest way to start running an open-source LLM locally?

Current guidance points to Ollama as the fastest option for prototyping locally, with workflow choices then depending on Apple unified-memory systems, one consumer GPU, or an enterprise deployment stack [19][33]. Start with Qwen3-32B from the Qwen Team at Alibaba if your hardware matches its quantized footprint and you need an Apache 2.0 license [1].

Sources

  1. Best Open-Source LLM Models in 2026: Coding, Local ... — 2025-11-12
  2. unsloth/DeepSeek-R1-Distill-Qwen-32B — 2026-08-26
  3. The Best Open Source and Open-Weight LLM Models to ... — 2026-05-13
  4. Models – Hugging Face — 2026-08-14
  5. joylarkin/AI-Coding-Models · Datasets at Hugging Face — 2026-05-24
  6. Vision Language Models (Better, faster, stronger) — 2025-05-12
  7. State of Open Source on Hugging Face: Spring 2026 — 2026-03-17
  8. The Best Open Source LLMs (2026): Ranked by Benchmark ... — 2026-06-18
  9. Large Language Model Category — 2026-09-22
  10. 200+ Roleplay, Creative Writing, Uncensored, NSFW models. — 2024-11-14
  11. moncefem/featherless-ai-models · Datasets at Hugging Face — 2026-05-19
  12. State of Open Models: Summer 2026 Observations — 2026-08-14
  13. Best Open Source LLMs in 2026: We Reviewed 7 Models — 2026-07-08
  14. The Future of Large Language Models - AIMultiple — 2026-09-17
  15. Best Open Source LLM in 2026: 10 Models Ranked - Atomic Chat — 2026-07-21
  16. The Best Open Source LLMs: 191 Models Compared - Gradually AI — 2024-04-26
  17. Best Local LLM 2026, the Models That Actually Run on Your Own Machine — 2026-09-02
  18. Best GPU for Local LLM Inference 2026 - Mayhemcode — 2026-06-18
  19. Best Open Source LLMs (September 2026) - Thunder Compute — 2026-05-27
  20. Best Consumer GPU for Local LLM 2026 — Buyer's Guide — 2026-04-29
  21. The Best Open Source LLM Guide 2026: Models - Mastra — 2026-08-08
  22. What to Buy for Local LLMs (April 2026) - Medium — 2026-04-03
  23. Enterprise Local LLM Deployment: vLLM, GPUs, Containers ... — 2026-03-20
  24. Best Open Source LLMs: Complete 2026 Guide - Contabo — 2026-01-09
  25. Best GPUs for Running Local LLMs (2026): VRAM, Bandwidth & Picks — 2026-06-06
  26. Best GPU for Local LLM — Buyer's Guide 2026 | Local AI Guide — 2026-07-12
  27. Best Open-Source LLMs in 2026: What Actually Runs ... — 2026-08-06
  28. Best Open-Source LLMs (2026): Free Models Ranked — 2026-02-04
  29. Best Open-Source LLMs: July 2026 Leaderboard | TECHSY — 2026-03-27
  30. Best Open Source LLM 2026: DeepSeek, Kimi, Qwen Ranked — 2026-06-03
  31. Best Local LLMs to Run in 2026: Ranked, With Specs — 2026-08-12
  32. Best Local LLMs by VRAM Tier 2026: 12GB, 24GB, 48GB Guide — 2026-04-01
  33. Best Local LLM for Coding in 2026: A Comprehensive Guide — 2026-08-13
  34. Best Open Source LLMs in 2026 - Onyx AI — 2026-05-05
  35. The Best Local LLM Models to Run in 2026 - daily.dev — accessed 2026-09-25
  36. Best Open Source Self-Hosted LLMs for Coding in 2026 | Pinggy Blog — 2026-03-26
  37. How to Choose the Best Open Source LLM for Your Project ... — 2025-09-09
  38. The Best Open-Source LLMs in 2026 - BentoML — 2023-05-29

Get these models on a hard drive

Skip the downloads. Browse our catalog of 985+ commercially-licensed AI models, available pre-loaded on high-speed drives.

Browse Model Catalog

More from the blog

Best Vision-Language Models to Run Locally in 2026
2026-09-25 · 11 min read
Best Quantization Formats in 2026: GGUF, AWQ, GPTQ, EXL2
2026-09-25 · 16 min read
Best Local AI Runtimes in 2026: llama.cpp, vLLM, Ollama
2026-09-25 · 18 min read
Best AI Models of July 2026: Complete Rankings by Benchmark, Use Case, and Price
2026-07-09 · 6 min read