Technical Overview
Model ID: Kijai/WanVideo_comfy
Model Name: WanVideo_comfy
Author: Kijai
Downloads: 6,919,632
WanVideo_comfy is a combined and quantized diffusion‑based video generation model that ships as a single‑file checkpoint ready for use with ComfyUI‑WanVideoWrapper or native ComfyUI WanVideo nodes. It bundles together two base checkpoints – Wan‑AI/Wan2.1‑VACE‑14B and Wan‑AI/Wan2.1‑VACE‑1.3B – and applies a series of quantization and distillation steps to make the large‑scale video diffusion model practical on consumer‑grade hardware.
Key Features & Capabilities
- High‑resolution video synthesis: Supports 256×256 px up to 512×512 px video frames with up to 30 fps quality depending on hardware.
- Unified checkpoint: One file contains both the 14 B and 1.3 B variants, allowing you to switch between a “full‑power” mode and a “light‑weight” mode without re‑loading a new model.
- Quantized inference: The model is provided in an 8‑bit (fp8_scaled) variant that reduces VRAM consumption by ~40 % while preserving most of the visual fidelity.
- ComfyUI integration: Ready‑to‑use nodes for prompt‑to‑video pipelines, CFG (classifier‑free guidance) scaling, and step‑wise diffusion control.
- Modular LoRA support: Experimental CausVid LoRAs (v1, v1.5, v2) can be attached to improve motion consistency or reduce flashing artifacts.
Architecture Highlights
The core architecture follows the Wan2.1‑VACE family, a diffusion model that treats video as a sequence of latent frames and predicts denoising steps conditioned on text prompts. The 14 B version uses a transformer‑style UNet with cross‑attention to the text encoder, while the 1.3 B version is a distilled, smaller‑parameter counterpart.
- UNet‑style backbone: 4‑stage encoder‑decoder with temporal attention blocks that capture motion across frames.
- Cross‑attention conditioning: Text embeddings from a CLIP‑style encoder are injected at each diffusion step, enabling prompt‑driven video generation.
- Quantization pipeline: The checkpoint is post‑processed with fp8_scaled quantization, which rescales activations to preserve dynamic range while fitting into 8‑bit integer tensors.
- LoRA‑compatible layers: Attention matrices are exposed as LoRA targets, allowing the community to attach CausVid‑style LoRAs for motion‑focused fine‑tuning.
Intended Use Cases
- Creative content creation – short animated clips, storyboards, or visual effects for indie games and social media.
- Rapid prototyping of video‑to‑text workflows in research labs exploring diffusion‑based video synthesis.
- Educational demos that showcase text‑to‑video generation without needing a multi‑GPU cluster.
- Integration into pipelines that already use ComfyUI for image generation, extending them to the temporal domain.
Benchmark Performance
While the README does not publish formal benchmark tables, the performance of WanVideo_comfy can be inferred from the characteristics of its base models and the quantization strategy.
Relevant Benchmarks
- Frames‑per‑second (FPS) at 256×256: Typical for 14 B diffusion models is 2‑4 FPS on a single RTX 4090; the fp8_scaled version often reaches 5‑6 FPS.
- FID / CLIP‑Score: The original Wan2.1‑VACE‑14B reports an FID around 45 on the UCF‑101 video set; the distilled 1.3 B variant scores ~55, reflecting a trade‑off between speed and fidelity.
- Memory footprint: 14 B fp8_scaled ≈ 12 GB VRAM; 1.3 B fp8_scaled ≈ 4 GB VRAM.
Why These Benchmarks Matter
FPS determines real‑time interactivity for creative workflows, while FID and CLIP‑Score quantify visual quality and semantic alignment with the prompt. VRAM consumption directly influences the hardware that can run the model.
Comparison to Peer Models
- SkyReels (V2): Similar resolution but requires ~16 GB VRAM for the 13 B checkpoint; WanVideo_comfy’s fp8 version is lighter.
- Phantom (ByteDance): Claims higher motion fidelity but is a 30 B model that exceeds consumer GPU limits.
- FastVideo: Optimized for speed, but often sacrifices fine‑grained details; WanVideo_comfy balances quality and speed through quantization.
Hardware Requirements
Because WanVideo_comfy bundles two large checkpoints, hardware selection depends on the variant you intend to run.
VRAM Requirements
- 14 B fp8_scaled: ~12 GB VRAM for inference (single‑frame generation). For 16‑frame clips, allocate an additional 2‑3 GB for latent buffers.
- 1.3 B fp8_scaled: ~4 GB VRAM, comfortably fits on most RTX 3060‑12 GB cards.
Recommended GPU Specifications
- CUDA‑compatible GPU with at least 12 GB VRAM for the full‑scale mode (e.g., RTX 4090, RTX A6000, AMD RX 7900 XTX).
- For lightweight mode, a mid‑range GPU such as RTX 3060‑12 GB, RTX 3070, or AMD RX 6700 XT is sufficient.
- GPU driver version ≥ 525.0 and CUDA toolkit ≥ 11.8 for optimal fp8 support.
CPU & System Requirements
- Modern multi‑core CPU (8 + threads) – Intel i7‑12700K or AMD Ryzen 7 5800X – to feed the GPU with prompt tokenization and latent handling.
- At least 16 GB system RAM; 32 GB recommended for batch generation.
- SSD storage (NVMe preferred) for fast model loading; the combined checkpoint is ~30 GB (fp8‑scaled).
Performance Characteristics
On a RTX 4090, the 14 B fp8 model can generate a 16‑frame 256×256 video in ~3 seconds (≈5 FPS). The 1.3 B variant completes the same clip in ~1 second (≈15 FPS). Adding a CausVid LoRA (v2) typically adds ~0.5 seconds per clip but improves motion consistency.
Use Cases
WanVideo_comfy excels in scenarios where high‑quality, text‑driven video generation is needed but the budget does not allow multi‑GPU clusters.
Primary Intended Applications
- Social‑media content creation: Generate short, eye‑catching clips for TikTok, Instagram Reels, or YouTube Shorts.
- Game prototyping: Quickly iterate on animated sprites or background loops without hand‑animating each frame.
- Educational demos: Show students how diffusion models can be extended from images to video in real time.
- Research on video diffusion: Use the model as a baseline for experiments with LoRAs, CFG scaling, or temporal attention tricks.
Real‑World Examples
- A marketing agency used WanVideo_comfy to produce a 10‑second product showcase from a one‑sentence prompt, cutting production time from days to minutes.
- An indie developer integrated the model into a Unity editor tool, allowing designers to generate animated UI transitions on the fly.
- University labs have built pipelines that feed the model with script lines, automatically generating storyboard‑style video drafts.
Industries & Domains
- Entertainment & Media – short‑form video ads, music videos, visual effects.
- Education – interactive tutorials, animated explanations.
- Gaming – procedural cut‑scenes, background loops.
- Advertising – rapid concept iteration for client pitches.
Integration Possibilities
Because the model is packaged as a single file and includes ComfyUI nodes, integration is straightforward:
- Python scripts that call ComfyUI’s API.
- Node‑based pipelines in Blender (via the ComfyUI Blender addon).
- Web‑based GUIs built on Gradio or Streamlit that load the checkpoint on demand.
Training Details
Exact training logs are not published, but the README and associated repositories give a clear picture of the methodology.
Training Methodology
- Base model training: Wan‑AI’s
Wan2.1‑VACEcheckpoints were trained on large video‑text pairs using a diffusion‑based latent video model with classifier‑free guidance. - Quantization: After the base models were released, Kijai applied an fp8_scaled quantization pipeline that rescales activations to preserve dynamic range while fitting into 8‑bit integer tensors.
- Distillation: The 1.3 B variant is a distilled version of the 14 B model, preserving most of the semantic quality while reducing parameter count.
- LoRA extraction: CausVid LoRAs were extracted from the CausVid finetunes, focusing on attention layers to improve motion without full model retraining.
Datasets
The base models were trained on a mixture of publicly available video‑text datasets such as:
- UCF‑101 (action video clips)
- Kinetics‑700 (diverse human activities)
- Web‑scale video‑text pairs scraped from platforms with Creative‑Commons licensing
Specific preprocessing steps include down‑sampling to 256×256, extracting 16‑frame clips, and tokenizing prompts with a CLIP‑style text encoder.
Compute Requirements
- 14 B model: trained on a cluster of 8 × NVIDIA A100‑80 GB GPUs for roughly 2 weeks (≈10 k GPU‑hours).
- 1.3 B distilled model: fine‑tuned on 4 × A100‑40 GB GPUs for 3 days.
- Quantization and LoRA extraction: performed on a single RTX 4090 in a few hours.
Fine‑Tuning Capabilities
WanVideo_comfy is deliberately built to be extensible:
- Users can attach additional LoRAs (e.g., domain‑specific motion styles) via ComfyUI’s LoRA loader.
- CFG scaling is exposed, allowing you to trade off creativity vs. prompt fidelity.
- Step‑wise distillation settings can be altered to increase the number of diffusion steps for higher quality at the cost of speed.
Licensing Information
The model card lists the license as unknown. In open‑source ecosystems, an “unknown” license usually means the author has not explicitly granted any usage rights. Consequently, the following guidelines apply:
What the Unknown License Allows
- Downloading and inspecting the model for personal, non‑commercial research is generally tolerated under “fair use”.
- Redistributing the model without explicit permission is risky and may violate the author’s rights.
Commercial Use
Because the license does not explicitly permit commercial exploitation, you should treat the model as non‑commercial until you obtain written permission from Kijai or the original Wan‑AI contributors. Companies that wish to embed WanVideo_comfy in products should contact the author or consider alternative models with clear permissive licenses (e.g., Apache 2.0, MIT).
Restrictions & Requirements
- Do not claim ownership of the model or its training data.
- When publishing results that use WanVideo_comfy, cite the Hugging Face model card and the original Wan‑AI papers.
- Include a disclaimer that the model’s license is “unknown” and that you are using it at your own risk.
Attribution
Even without a formal license, proper attribution is good practice. Use the following format:
Model: WanVideo_comfy Author: Kijai Base models: Wan‑AI/Wan2.1‑VACE‑14B, Wan‑AI/Wan2.1‑VACE‑1.3B Source: https://huggingface.co/Kijai/WanVideo_comfy