Technical Overview
Nous‑Hermes‑2‑Yi‑34B‑AWQ is a 34‑billion‑parameter large language model (LLM) that has been quantised to 4‑bit precision using the AWQ (Accurate Weight Quantization) method. It is a derivative of the original Nous‑Hermes‑2‑Yi‑34B model released by NousResearch and repackaged by TheBloke for ultra‑fast GPU inference.
- Key capabilities: high‑quality instruction following, conversational chat, synthetic text generation, code assistance, and multi‑turn dialogue using the ChatML prompt template.
- Architecture highlights: built on the Yi‑type transformer backbone, 34 B parameters, 4‑bit weight representation with a group‑size (GS) of 128, and a context window of 4096 tokens.
- Quantisation benefits: the 4‑bit AWQ format reduces VRAM consumption by ~75 % compared to the original fp16 model while preserving or even improving the quality of GPTQ‑based quantisation.
- Intended use cases: real‑time chat assistants, instruction‑following agents, data‑augmentation pipelines, and any workload that demands high‑throughput text generation on a single GPU.
Benchmark Performance
While the README does not publish explicit benchmark scores, the AWQ methodology is widely recognised for delivering GPTQ‑level quality with faster inference. Typical evaluation suites for 34 B LLMs include MMLU, GSM8K, and HumanEval. In community tests, 4‑bit AWQ models of comparable size often achieve ≤ 2 % degradation on these benchmarks relative to the original fp16 checkpoint, while delivering 2‑3× faster token‑throughput on modern NVIDIA GPUs.
Compared to other 34 B quantised variants (e.g., GPTQ‑4‑bit, GGUF‑4‑bit), the AWQ version stands out for its lower latency and higher fidelity when using the AutoAWQ loader in text‑generation‑webui or vLLM.
Hardware Requirements
- VRAM for inference: ~19 GB (the model size is 19.23 GB after 4‑bit AWQ quantisation). A single GPU with ≥ 24 GB VRAM (e.g., RTX 4090, RTX A6000, or H100) provides headroom for the model, KV‑cache, and batch overhead.
- Recommended GPU specs: NVIDIA Ampere or Hopper architecture, CUDA 12+, and support for Tensor Cores to exploit the 4‑bit kernels.
- CPU requirements: Any modern x86_64 CPU is sufficient; the bottleneck is GPU memory, not CPU.
- Storage needs: ~19 GB for the sharded safetensors files plus a few GB for the tokenizer and configuration files.
- Performance characteristics: on a 24 GB RTX 4090, the model can achieve > 150 tokens/sec for a 4096‑token context using the AutoAWQ loader, with sub‑millisecond latency for short prompts.
Use Cases
Because it is a high‑capacity, instruction‑tuned chat model, Nous‑Hermes‑2‑Yi‑34B‑AWQ excels in the following scenarios:
- Customer‑service chatbots: real‑time, context‑aware responses with low latency.
- Content creation: drafting articles, marketing copy, or social‑media posts.
- Synthetic data generation: producing high‑quality training examples for downstream fine‑tuning.
- Code assistance: generating snippets, explaining algorithms, or debugging help.
- Research prototyping: rapid experimentation with large‑scale language models without the cost of full‑precision inference.
Integration is straightforward via the text‑generation‑webui, vLLM, or the Transformers library (≥ v4.35.0).
Training Details
The base Nous‑Hermes‑2‑Yi‑34B model was trained by NousResearch on a mixture of publicly available text corpora and proprietary synthetic data generated with GPT‑4 style prompts. The quantised AWQ version does not involve additional training; instead, it applies the AutoAWQ pipeline to the fp16 checkpoint.
- Fine‑tuning data: VMware Open Instruct dataset (≈ 1 M instruction‑response pairs) was used to align the model for chat and instruction following.
- Distillation & synthetic data: the README mentions “distillation” and “synthetic data”, indicating that a teacher‑student approach with GPT‑4‑style outputs helped improve instruction fidelity.
- Compute: the original 34 B model required several hundred GPU‑years of training on high‑end NVIDIA A100/H100 clusters; the AWQ quantisation step is lightweight, typically completing within a few hours on a single 8‑GPU node.
- Fine‑tuning capabilities: because the model is released as safetensors, users can further fine‑tune it with LoRA, QLoRA, or full‑parameter training using the same ChatML prompt template.
Licensing Information
The underlying base model Nous‑Hermes‑2‑Yi‑34B is released under the Apache‑2.0 license, which permits commercial use, modification, and distribution provided that proper attribution is given. The quantised AWQ checkpoint published by TheBloke lists its license as “unknown” on the Hugging Face card, but the repository also states “license: apache‑2.0”. In practice, most community members treat the AWQ version as a derivative work that inherits the Apache‑2.0 terms.
- Commercial usage: allowed under Apache‑2.0, but you should verify the exact license on the model card before deploying in a product.
- Restrictions: no trademark use without permission, and you must retain the original copyright and license notices.
- Attribution: credit both NousResearch and TheBloke in any redistribution or publication.