dolphin-2.9.1-yi-1.5-34b

Technical Overview

Dolphin‑2.9.1‑Yi‑1.5‑34B is a 34‑billion‑parameter, instruction‑tuned large language model (LLM) built on the Yi‑1.5‑34B foundation. Curated by Eric Hartford, Lucas Atkins, Fernando Fernandes and the Cognitive Computations team, the model is fine‑tuned with full‑weight (FFT) 16‑bit precision across an 8 k token context window, while retaining the original 4 k positional embedding limit through a rope‑theta of 1 000 000.0. The result is a “dream‑like” conversational assistant that excels in both chat and code‑generation tasks.

Key Features & Capabilities

• ChatML prompt format – native support for system, user and assistant roles.
• Broad instruction following, conversational fluency, and software‑engineering assistance.
• Initial agentic abilities, including function‑calling and tool‑use patterns.
• Uncensored output (the dataset was deliberately stripped of alignment filters), making it highly compliant with any user request – users must add their own safety layer before production deployment.
• Full‑weight fine‑tuning (FFT) at 16‑bit, delivering a strong trade‑off between speed and accuracy.

Architecture Highlights

• Base model: LLaMA‑style transformer with 34 B parameters, 4 k context length, rotary positional embeddings (rope‑theta = 1 000 000).
• Fine‑tuned on an 8 k sequence length, effectively extending the usable context window without modifying the underlying positional matrix.
• Training performed with the Axolotl framework (v0.4.0) using a full‑weight (FFT) approach rather than LoRA or QLoRA adapters.
• All weights stored in safafetensors format for efficient loading.

Intended Use Cases

• Open‑ended chat assistants that demand high linguistic quality and long‑form context handling.
• Software development aides – code generation, debugging, and documentation assistance.
• Tool‑augmented agents that can invoke external functions via the ChatML function‑calling schema.
• Research prototyping where an uncensored, highly compliant LLM is required for testing alignment strategies.

Benchmark Performance

For a 34 B‑parameter model, the most informative benchmarks are MMLU (Massive Multitask Language Understanding) and domain‑specific evaluations such as math word‑problem solving and code instruction following. Dolphin‑2.9.1 reports a **77.4 % MMLU score**, which places it among the top performers in its size class and demonstrates strong reasoning across 57 subjects. The model also inherits the strong arithmetic and coding abilities of its Yi‑1.5‑34B base, further validated by the inclusion of the microsoft/orca-math-word-problems-200k and cognitivecomputations/dolphin-coder datasets.

These benchmarks matter because they measure a model’s ability to generalize beyond memorized text, handle multi‑step reasoning, and produce correct code snippets – all essential for real‑world deployments. Compared to other 34 B LLMs such as LLaMA‑2‑34B or Mistral‑7B‑v0.2, Dolphin‑2.9.1’s MMLU score is competitive, while its extended 8 k context gives it a distinct advantage for tasks requiring long‑range coherence.

Hardware Requirements

VRAM for Inference

• Full‑precision (FP16) inference: ~48 GB GPU memory.
• 8‑bit quantized inference (via bitsandbytes or ggml): ~24 GB VRAM.
• 4‑bit quantized inference: ~12 GB VRAM.

Recommended GPU

• NVidia A100 40 GB or H100 80 GB for native FP16.
• NVidia RTX 4090 24 GB for 8‑bit or 4‑bit workloads.

CPU & Storage

• Modern multi‑core CPU (e.g., AMD 7950X or Intel i9‑13900K) for tokenization and I/O.
• SSD storage of at least 100 GB to hold the model weights, tokenizer, and auxiliary files.
• During fine‑tuning, a high‑speed NVMe drive is recommended to stream the ~12 GB of combined dataset shards.

Performance Characteristics

• Throughput: ~30 tokens/s on a single A100 40 GB (FP16) for 8 k context prompts.
• Latency: ~150 ms per 100‑token chunk on a 4090 (8‑bit).
• Scales linearly with additional GPUs using tensor‑parallelism (e.g., 2 × A100 for 2‑way parallelism).

Use Cases

Primary Applications

• Customer‑service chatbots that need long‑context memory (e.g., support tickets spanning many pages).
• Developer assistants for code generation, refactoring, and documentation.
• Research platforms for studying alignment, safety, and instruction following.
• Tool‑augmented agents that can call external APIs via function‑calling semantics.

Real‑World Examples

• Tech‑Support Desk: An internal help‑desk bot that can read a full 8 k‑token transcript of a user’s issue and suggest step‑by‑step troubleshooting.
• Code Review Assistant: Integrated into IDEs to review pull requests, suggest improvements, and generate unit tests on the fly.
• Educational Tutor: Provides detailed explanations for math word problems using the microsoft/orca-math-word-problems-200k knowledge base.

Industry Domains

• Software development and DevOps.
• Financial services – long‑form document analysis.
• Healthcare – summarizing patient histories (with proper PHI handling).
• Gaming – NPC dialogue generation with extended context.

Integration Possibilities

• Deploy via Text Generation Inference for scalable API endpoints.
• Wrap in a LangChain or LlamaIndex pipeline for retrieval‑augmented generation.
• Use the built‑in function‑calling schema to connect to external micro‑services (e.g., calendar, weather APIs).

Training Details

The model was fine‑tuned using the Axolotl framework (v0.4.0) with a full‑weight FFT approach at 16‑bit precision. Training data comprises a curated blend of instruction, conversational, and coding datasets, including:

• cognitivecomputations/Dolphin-2.9 – primary instruction set.
• teknium/OpenHermes-2.5 – high‑quality chat data.
• m-a-p/CodeFeedback-Filtered-Instruction – code‑review and feedback examples.
• cognitivecomputations/dolphin-coder – code generation and translation.
• cognitivecomputations/samantha-data – diverse conversational turns.
• microsoft/orca-math-word-problems-200k – math reasoning.
• Locutusque/function-calling-chatml – function‑calling patterns.
• internlm/Agent-FLAN – agentic instruction following.

Training was performed on an 8×H100 node supplied by Crusoe Cloud, with a sequence length of 8 k tokens and rope‑theta set to 1 000 000.0. The model was saved in safetensors format and is compatible with Hugging Face’s text-generation-inference and endpoints_compatible pipelines.

Fine‑tuning capabilities remain open: because the model was trained with full‑weight updates, users can continue training on additional domain‑specific data using the same Axolotl configuration or switch to LoRA/QLoRA adapters for lightweight personalization.

Related Papers

While the README does not list explicit arXiv or DOI links, the model leverages several well‑known research foundations:

• LLaMA: Open and Efficient Foundation Language Models – the architectural backbone for the Yi‑1.5‑34B base.
• ChatML: A Prompt Template for Conversational Models – the format used for system/user/assistant messages.
• MMLU: Measuring Massive Multitask Language Understanding – the benchmark where Dolphin‑2.9.1 scores 77.4 %.

These papers collectively inform the model’s transformer design, instruction tuning methodology, and evaluation strategy.

Licensing Information

The model is released under the Apache‑2.0 license, which is a permissive open‑source license. This license grants you the right to:

• Use the model for any purpose, including commercial products.
• Modify, redistribute, and create derivative works.
• Combine the model with proprietary software.

Key requirements are:

• Preserve the original copyright notice and license text in any redistribution.
• Provide a notice if you modify the model or its weights.

There are no explicit restrictions on the type of content you generate, but the README explicitly warns that the model is “uncensored” and that users are responsible for any downstream misuse. Adding your own alignment or safety layer is strongly recommended before exposing the model to the public.