Qwen2.5-Coder-1.5B-Instruct-AWQ

Technical Overview

What is this model? Qwen2.5‑Coder‑1.5B‑Instruct‑AWQ is a 1.54 billion‑parameter, instruction‑tuned large language model specialized for software development tasks. It is the 4‑bit AWQ‑quantized variant of the Qwen2.5‑Coder‑1.5B‑Instruct base model, designed to run efficiently on commodity GPUs while retaining strong code‑generation, reasoning, and fixing abilities.

Key features & capabilities

• Optimized for code generation, code reasoning and code fixing across 20+ programming languages.
• Supports full‑context windows of up to 32,768 tokens, enabling multi‑file or long‑script interactions.
• AWQ 4‑bit quantization reduces VRAM footprint to roughly 5 GB while preserving near‑full‑precision quality.
• Instruction‑tuned with a conversational format (system/user/assistant messages) for seamless integration into IDE assistants, code‑review bots, and autonomous code agents.
• Built on the Qwen2.5 foundation: RoPE positional encoding, SwiGLU activation, RMSNorm, and GQA (12 Q‑heads, 2 KV‑heads) for fast attention.

Architecture highlights

• 28 transformer layers with Grouped‑Query Attention (GQA) – 12 query heads, 2 key/value heads per layer.
• RMSNorm for stable training at large sequence lengths.
• SwiGLU activation, which has been shown to improve performance on code‑related tasks.
• Word‑embedding tying reduces parameter count and improves generalisation.

Intended use cases The model is meant for developers and enterprises that need a lightweight yet high‑quality code‑LLM for:

• Auto‑completion and snippet generation inside IDEs.
• Automated bug‑fix suggestions and refactoring.
• Code‑review assistants that can explain or critique pull‑requests.
• Autonomous “code agents” that can plan, write, and test software without human intervention.

Benchmark Performance

The Qwen2.5‑Coder family is evaluated on a suite of code‑centric benchmarks (HumanEval, MBPP, CodeAlpaca, and the Qwen2.5‑Coder technical report). The 1.5 B‑Instruct‑AWQ variant achieves:

• HumanEval pass@1 ≈ 48 % (near‑parity with the full‑precision 1.5 B model).
• MBPP pass@1 ≈ 55 %, outperforming many open‑source 2 B‑scale models.
• Inference throughput of ~120 tokens/s on an NVIDIA A100 (40 GB) with 4‑bit AWQ, and ~45 tokens/s on a RTX 3080 (10 GB) when using device_map="auto".

These benchmarks matter because they directly measure a model’s ability to synthesize correct, runnable code from natural‑language prompts—a core requirement for IDE assistants and autonomous coding agents. Compared to similar 1‑2 B‑scale models (e.g., CodeLlama‑7B‑Instruct‑Quantized, DeepSeek‑Coder‑1.3B), Qwen2.5‑Coder‑1.5B‑AWQ delivers higher pass rates while using less VRAM, making it a sweet spot for edge‑deployment scenarios.

Hardware Requirements

VRAM for inference The 4‑bit AWQ quantization brings the model size to roughly 5 GB of GPU memory (including the transformer weights and KV cache for a 32 k context). A GPU with at least 8 GB VRAM is recommended to allow head head and batch‑size flexibility.

• Recommended GPUs: NVIDIA RTX 3080/3090, RTX 4090, A100 (40 GB), or any GPU supporting torch.float16 and bitsandbytes for AWQ.
• CPU: Modern x86_64 or ARM64 CPUs; 8‑core CPUs are sufficient for tokenization and I/O, but the heavy lifting stays on the GPU.
• Storage: Model files (safetensors + tokenizer) total ~2 GB. SSD storage is recommended for fast loading.
• Performance notes: With device_map="auto", the model will automatically offload parts of the transformer to CPU if GPU memory is limited, at the cost of reduced throughput.

Use Cases

Primary applications revolve around software development assistance:

• IDE auto‑completion: Plug‑in for VS Code, JetBrains, or Vim to suggest whole functions or refactorings.
• Code‑review bots: Automated reviewers that can explain why a PR fails tests and suggest fixes.
• Educational tools: Interactive tutoring systems that generate example code and explain concepts.
• Autonomous agents: Agents that can plan a feature, write the code, run unit tests, and iterate without human input.

Industries that benefit include:

• Software engineering & DevOps
• FinTech (automated compliance scripts)
• EdTech (coding courses and labs)
• Gaming (script generation for Unity/Unreal)

Integration is straightforward via the transformers library or the Text Generation Inference server, which supports the text-generation pipeline tag.

Training Details

Methodology The model underwent two phases:

• Pre‑training: 5.5 trillion tokens drawn from a mixture of web text, open‑source repositories, and synthetic code‑generation data. The training used the Qwen2.5 transformer backbone with RoPE positional embeddings.
• Instruction‑tuning (post‑training): Fine‑tuned on a curated instruction set that emphasizes code‑specific prompts (e.g., “write a quick‑sort”, “fix the bug in this snippet”). The fine‑tuning dataset includes ~1 billion instruction‑response pairs, many of which are code‑centric.

Datasets include:

• GitHub public code (multiple languages)
• StackOverflow Q&A pairs
• Synthetic code‑generation data produced by earlier Qwen models
• General‑purpose text corpora for maintaining broader language competence

Compute Training was performed on clusters of NVIDIA A100 GPUs (40 GB) with mixed‑precision (FP16) and gradient checkpointing to fit the 1.5 B‑parameter model. The exact FLOP count is not disclosed, but the scale is comparable to other 1‑2 B‑parameter LLMs (≈ 200 PF‑days).

Fine‑tuning capabilities The model can be further instruction‑tuned on domain‑specific codebases using LoRA or QLoRA, thanks to its standard transformers interface. The 4‑bit AWQ quantization can be retained during fine‑tuning with the bitsandbytes library to keep memory usage low.

Related Papers

The model builds on two arXiv pre‑prints:

• Qwen2.5‑Coder Technical Report – introduces the Qwen2.5‑Coder family, training methodology, and benchmark results.
• Qwen2 Technical Report – provides the foundation architecture (RoPE, SwiGLU, RMSNorm) that Qwen2.5‑Coder inherits.

These papers detail the 5.5 trillion‑token training corpus (including source code, synthetic data, and text‑code grounding) and the design decisions that enable state‑of‑the‑art code generation at the 1‑2 B‑parameter scale.

Licensing Information

The model is released under the Apache‑2.0 license. This permissive license permits:

• Free use, modification, and distribution for both commercial and non‑commercial purposes.
• Inclusion in proprietary products, provided that the original copyright notice and license text are retained.
• Patents granted by the contributors are also covered, reducing legal risk for commercial deployments.

There are no “unknown” restrictions; the Apache‑2.0 license explicitly allows commercial exploitation. The only requirement is proper attribution (e.g., citing the technical report and linking to the model card). If you redistribute the model, you must include the same license file.