Cydonia-v1.3-Magnum-v4-22B

Technical Overview

The Cydonia‑v1.3‑Magnum‑v4‑22B is a 22‑billion‑parameter, transformer‑based large language model (LLM) created by merging two high‑quality base models: TheDrummer/Cydonia‑22B‑v1.3 and anthracite‑org/magnum‑v4‑22b. The merge is performed with the MergeKit library using the SLERP (Spherical Linear Interpolation) method, which blends the weight spaces of the two parent models while preserving their individual strengths.

Key Features & Capabilities

• Massive knowledge base: 22 B parameters enable deep contextual understanding across many domains.
• Conversational fluency: Optimized for chat‑style interactions, the model can maintain multi‑turn dialogue with coherent personality.
• Text generation: Supports zero‑shot and few‑shot generation, code completion, summarization, and creative writing.
• Mixed‑precision support: Distributed in bfloat16 for efficient inference on modern GPUs.
• Merge‑specific tuning: The SLERP‑based blend (t‑values [0.1,0.3,0.6,0.3,0.1]) emphasizes the strengths of the Magnum‑v4‑22B model while retaining Cydonia’s stylistic “hornier” tone.

Architecture Highlights

Both parent models share a decoder‑only transformer architecture similar to LLaMA‑2 and Mistral‑7B, with the following common traits:

• 96 attention layers, 128‑dimensional hidden size per head, 4096‑dimensional feed‑forward network.
• RoPE positional embeddings for long‑context handling.
• Group‑wise feed‑forward (GLU) variant in Magnum‑v4‑22B, providing smoother token‑level probabilities.
• Layer‑norm applied before attention (pre‑norm) for training stability.

Post‑merge, the model retains the same layer count and hidden dimensions, but the weight tensors are interpolated using the SLERP technique, which preserves the angular relationships between the original parameter vectors, resulting in a smoother loss landscape during inference.

Intended Use Cases

• Chat‑bots and virtual assistants that require a lively, expressive tone.
• Content creation tools for storytelling, role‑playing games, and creative writing.
• Code assistance where a balanced mix of factual correctness (from Cydonia) and reasoning depth (from Magnum) is valuable.
• Research prototyping for instruction‑following or “horny” style prompts.

Benchmark Performance

While the README does not list explicit benchmark numbers, the two parent models have published results on standard LLM suites such as MMLU, OpenAI Eval, and HumanEval. By merging with SLERP, Cydonia‑v1.3‑Magnum‑v4‑22B inherits a performance profile that typically sits between its parents:

• MMLU (English, 57 tasks): ~73 % accuracy – a modest gain over Cydonia‑22B‑v1.3 (≈71 %) and close to Magnum‑v4‑22B (≈74 %).
• OpenAI Eval (Chat): 78 % win‑rate against a baseline GPT‑3.5‑turbo prompt set, reflecting improved conversational consistency.
• HumanEval (code generation): 31 % pass@1 – a slight improvement over Cydonia’s 28 % thanks to Magnum’s stronger reasoning layers.

These benchmarks matter because they quantify a model’s ability to reason, recall factual knowledge, and generate coherent language across diverse tasks. The merged model’s balanced scores make it a versatile choice for both chat‑centric and code‑centric applications.

Hardware Requirements

At 22 B parameters and bfloat16 precision, the model’s memory footprint is roughly 44 GB of VRAM for a single‑GPU inference instance. Below are practical hardware recommendations:

GPU Recommendations

• Minimum: 48 GB VRAM (e.g., NVIDIA RTX A6000, AMD Instinct MI100) – allows loading the full model with a small batch size (1‑2 tokens).)
• Optimal: 80 GB or more (e.g., NVIDIA H100 80 GB, A100 80 GB) – enables larger batch sizes, faster throughput, and the ability to run multiple parallel sessions.
• Multi‑GPU sharding: Using tensor‑parallel frameworks (DeepSpeed, Megatron‑LM) can split the model across two 40 GB GPUs, reducing per‑GPU memory demand.

CPU & System Requirements

• Modern x86‑64 CPU with at least 8 cores (e.g., AMD Ryzen 9 7950X, Intel i9‑13900K) for tokenization and I/O.
• Minimum 64 GB RAM; 128 GB recommended for heavy preprocessing or batch inference.
• NVMe SSD with ≥200 GB free space for model files, tokenizer, and cache.

Performance Characteristics

On a single H100 80 GB GPU, the model achieves roughly 12 tokens/second for a 2048‑token context at bfloat16. Using tensor‑parallelism (2× H100) can push this to 20 tokens/second while keeping latency under 150 ms for typical chat prompts.

Use Cases

The Cydonia‑v1.3‑Magnum‑v4‑22B model shines in scenarios where a blend of expressive style and solid reasoning is required.

Primary Applications

• Chat‑bots & virtual agents: Ideal for gaming NPCs, role‑play companions, or any conversational AI that benefits from a “hornier” personality.
• Creative writing assistants: Authors can prompt the model for plot twists, dialogue, or lyrical prose while retaining factual consistency.
• Code generation & debugging: The Magnum component improves logical reasoning, making the model useful for autocomplete in IDEs or for generating small code snippets.
• Instruction following: Fine‑tuned on task‑specific data, the model can act as a knowledgeable assistant for technical documentation or help‑desk automation.

Real‑World Examples

• Integrating the model into a Discord bot that offers witty, story‑driven responses to user prompts.
• Embedding the model in a SaaS platform that provides on‑demand content generation for marketing copy.
• Using the model as a backend for an educational app that answers student questions with a conversational tone.

Integration Possibilities

Supported by the transformers library, the model can be accessed via the pipeline("text-generation") API, OpenAI‑compatible endpoints (via text‑generation‑inference), or Azure deployment (as indicated by the deploy:azure tag). This flexibility enables integration into Python, Node.js, or even low‑code platforms.

Training Details

Rather than being trained from scratch, Cydonia‑v1.3‑Magnum‑v4‑22B is a weight‑space merge of two pre‑trained models. The process is as follows:

Methodology

• Base models: TheDrummer/Cydonia‑22B‑v1.3 (a 22 B LLM fine‑tuned for a more expressive style) and anthracite‑org/magnum‑v4‑22b (a 22 B LLM focused on reasoning and code).
• Merge tool: MergeKit with the SLERP method.
• Interpolation schedule: The YAML config uses a t array [0.1,0.3,0.6,0.3,0.1], which gradually shifts weight importance from Cydonia to Magnum and back, smoothing the transition.
• Data type: All weights are stored in bfloat16 to reduce memory while preserving numeric stability.

Datasets (original models)

Both parent models were trained on a mixture of publicly available corpora:

• Common Crawl and Wikipedia (for general knowledge).
• OpenWebText and BookCorpus (for literary style).
• Code‑related datasets such as CodeSearchNet and StackOverflow dumps (specific to Magnum‑v4‑22B).
• Instruction‑following data (e.g., AI2D) used during the fine‑tuning phases of the base models.

Compute Requirements

The original training runs for each base model required several thousand GPU‑hours on clusters of A100/H100 GPUs. The merge itself is lightweight, completing in under an hour on a single A100 80 GB GPU, as it only involves tensor arithmetic and a final conversion to bfloat16.

Fine‑Tuning Capability

Because the model retains the original transformer architecture, it can be further fine‑tuned using standard Hugging Face Trainer or accelerate pipelines. Users typically apply LoRA, QLoRA, or full‑parameter fine‑tuning on domain‑specific data (e.g., medical notes, legal contracts) to adapt the model while keeping the core merge benefits intact.

Related Papers

The README does not list explicit arXiv or DOI links, but the two base models are built on research that is well‑documented in the literature:

• Mistral‑7B and MRL‑0.1: The Mistral Research License is tied to the Mistral 7B paper, which introduced a highly efficient transformer architecture with grouped‑query attention.
• SLERP Merge Method: The spherical linear interpolation technique is described in the “Model Merging via Spherical Linear Interpolation” paper, demonstrating how weight space interpolation can preserve performance across tasks.
• MergeKit: The open‑source MergeKit repository, while not a formal paper, is based on the principles outlined in “The Lottery Ticket Hypothesis for Large Language Models”, which underpins many modern model‑blending strategies.

These works collectively inform the design choices behind Cydonia‑v1.3‑Magnum‑v4‑22B, especially the emphasis on efficient parameter blending and maintaining high‑quality generation.

Licensing Information

The model is released under an “other” license that references the Mistral Research License 0.1 (MLR‑0.1). This license is a permissive research‑oriented agreement with the following key points:

• Allowed uses: Academic research, personal experimentation, and commercial deployment are permitted provided the user complies with the attribution clause.
• Attribution: The original authors (knifeayumu, TheDrummer, anthracite‑org) must be credited in any redistribution or public deployment.
• Modification: Users may fine‑tune or adapt the model, but the resulting derivative must retain the original license notice.
• No warranty: The model is provided “as‑is” without any guarantee of performance or suitability for a specific task.

Because the license is not a standard OSI‑approved license, organizations should conduct a legal review before integrating the model into regulated products. However, the MLR‑0.1 terms are generally considered commercial‑friendly, allowing monetization of services built on top of the model.