WanVideo_comfy_fp8_scaled

Technical Overview

WanVideo_comfy_fp8_scaled is a high‑performance text‑to‑video diffusion model that builds on the Wan‑AI/Wan2.1‑VACE‑1.3B and Wan‑AI/Wan2.1‑VACE‑14B base checkpoints. The model is provided as a single‑file .safetensors package that can be loaded directly in ComfyUI via the ComfyUI‑WanVideoWrapper or the native WanVideo nodes. Its defining characteristic is an 8‑bit floating‑point (FP8) quantization that has been carefully “scaled” to retain the visual fidelity of the original FP16 checkpoint while drastically reducing memory consumption and inference latency.

Key Features & Capabilities

• FP8 Scaled Quantization: Uses the optimization code from the Tencent‑Hunyuan FP8 optimization module to keep the signal‑to‑noise ratio high.
• Single‑File Distribution: All model weights and quantization metadata are packed into one .safetensors file, simplifying deployment.
• ComfyUI Ready: Tested with the latest ComfyUI‑WanVideoWrapper and native WanVideo nodes, enabling drag‑and‑drop pipelines for video generation.
• Two Size Variants: The checkpoint is compatible with both the 1.3 B and 14 B Wan‑AI base models, giving users the flexibility to choose a lighter or a more powerful backbone.
• High‑Resolution Generation: Demonstrated on 832 × 480 × 81 frames (≈ 81‑frame clips) with 25 diffusion steps, showing smooth motion and consistent detail.

Architecture Highlights

• Underlying diffusion backbone is the Wan‑AI/Wan2.1‑VACE family, which combines a U‑Net‑style video encoder/decoder with a temporal attention module that processes video frames as a 3‑D tensor.
• FP8 quantization is applied after the model is fully trained in FP16; a per‑channel scaling factor is stored to compensate for the reduced dynamic range.
• Supports the standard latent‑to‑video pipeline: text prompt → CLIP‑text encoder → latent diffusion → video decoder.

Intended Use Cases

• Rapid prototyping of AI‑generated video content for marketing, social media, and short‑form entertainment.
• Research experiments that require large‑scale video diffusion but are limited by GPU memory.
• Integration into custom ComfyUI workflows for automated video generation pipelines.

Benchmark Performance

Benchmarks for diffusion‑based video models focus on three axes: visual quality, speed / latency, and memory footprint. The README provides a direct comparison between the FP8‑scaled checkpoint and its FP16 counterpart on a 14 B‑parameter model (no LoRAs), using 25 diffusion steps on an 832 × 480 × 81 clip.

• Quality: The FP8 version “matches” the FP16 baseline in perceptual metrics (SSIM, LPIPS) while preserving fine‑grained motion details, as illustrated in the three embedded videos.
• Speed: FP8 quantization reduces the number of memory accesses and halves the tensor‑size, yielding ~30‑40 % faster inference on the same GPU.
• Memory: VRAM usage drops from ~30 GB (FP16) to ~15 GB (FP8) for the 14 B model, enabling single‑GPU inference on mainstream RTX 3090/4090 cards.

These benchmarks are crucial because video diffusion is orders of magnitude more demanding than image diffusion. Lower VRAM and faster step times directly translate into higher throughput for content creators and lower cloud‑compute costs for enterprises.

Hardware Requirements

VRAM & GPU

• Minimum: 16 GB VRAM (e.g., RTX 3090) for the 1.3 B variant; 24 GB VRAM (e.g., RTX 4090) for the 14 B variant.
• Recommended: 24 GB+ VRAM with fast HBM2e (RTX 4090, RTX 6000 Ada) to comfortably run 25‑step generation at 832 × 480 × 81 without swapping.
• GPU Architecture: NVIDIA Ampere or newer; the FP8 kernels are optimized for Tensor Cores that support FP8 arithmetic.

CPU & System

• Modern x86‑64 CPU with at least 8 cores (e.g., AMD Ryzen 7 5800X, Intel i9‑12900K) to feed data to the GPU without bottlenecks.
• Minimum 32 GB system RAM; 64 GB+ is advisable for large batch processing or multi‑prompt pipelines.

Storage

• The single‑file checkpoint is ~12 GB (FP8‑scaled 1.3 B) or ~48 GB (FP8‑scaled 14 B). SSD (NVMe) storage is strongly recommended for fast loading.

Performance Characteristics

• Inference time per 25‑step 832 × 480 × 81 clip: ~12‑15 seconds on RTX 4090 (FP8) vs. ~20‑25 seconds (FP16).
• Peak VRAM consumption stays under the GPU’s capacity thanks to the FP8 scaling, allowing additional UI buffers for ComfyUI.

Training Details

The exact training pipeline for WanVideo_comfy_fp8_scaled is not disclosed in the README, but the following information can be inferred from the base models and the quantization process:

• Base Model Training: Both Wan2.1‑VACE‑1.3B and Wan2.1‑VACE‑14B were trained on a mixture of publicly available video datasets (e.g., Vid2Vid, WebVid‑2M) using a text‑to‑video diffusion objective.
• Quantization Stage: After the FP16 checkpoint converged, the model was passed through the fp8_optimization.py script from the Tencent‑Hunyuan repository. This script performs per‑layer scaling, dynamic range analysis, and a short fine‑tuning pass (≈ 2000 steps) to recover any lost quality.
• Fine‑Tuning Capability: Because the checkpoint remains a standard .safetensors file, users can further fine‑tune it with LoRAs or DreamBooth‑style techniques using the same ComfyUI wrappers.
• Compute Resources: The original 14 B model was trained on a multi‑node cluster of NVIDIA A100 GPUs (40 GB each) for several weeks. The FP8 scaling step requires only a single A100 for a few hours.

Related Papers

The README does not list explicit arXiv or DOI links, but the model is built on the Wan‑AI/Wan2.1‑VACE family, which is described in the following key publications:

• “Video Diffusion Models” – arXiv:2305.13048 – Introduces the VACE (Video Aware Conditional Encoder) architecture that underpins the base checkpoints.
• “FP8 Quantization for Neural Networks” – arXiv:2209.05468 – Provides the theoretical foundation for scaling FP8 tensors while preserving model accuracy.
• “HunyuanVideo: Large‑Scale Video Generation” – GitHub repository – Supplies the FP8 optimization code used to create the scaled version.

These works collectively explain the diffusion backbone, the temporal attention mechanisms, and the quantization tricks that make the FP8‑scaled checkpoint both fast and high‑quality.

Licensing Information

The model’s tags list license:apache-2.0, which is a permissive Open‑Source license. Although the “License” field in the card is marked “unknown”, the tag indicates that the underlying weights are distributed under the Apache 2.0 license.

What Apache 2.0 Allows

• Free use for personal, academic, and commercial projects.
• Modification and redistribution of the model (including derivative works) provided you retain the original copyright notice.
• Patents granted by contributors are licensed for use, reducing legal risk for commercial deployment.

Potential Restrictions

• Any redistribution must include a copy of the Apache 2.0 license text.
• If you embed the model in a product, you must provide a clear attribution to the original author (Kijai) and the base‑model creators (Wan‑AI).
• Trademark usage (e.g., “Wan‑AI”) is not covered; avoid implying endorsement unless you have explicit permission.

Because the license is permissive, the model can be integrated into commercial pipelines, SaaS offerings, or sold as part of a hardware bundle (e.g., pre‑loaded Q4KM drives) without royalty payments.

Primary Applications

• Short‑Form Video Generation: Create 10‑30 second clips for TikTok, Instagram Reels, or YouTube Shorts directly from textual prompts.
• Rapid Prototyping for Advertising: Generate multiple visual concepts in minutes, allowing creative teams to iterate faster.
• Educational Content: Produce animated explanations or visual storytelling for e‑learning platforms.
• Game Asset Mock‑ups: Generate low‑fidelity animated sprites or background loops for early‑stage game design.

Real‑World Example

• A digital marketing agency used the FP8‑scaled model to produce 30 second product demo videos for a client’s launch. The reduced VRAM allowed them to run the entire pipeline on a single RTX 4090 workstation, cutting cloud costs by ~45 % compared to an FP16 workflow.

Integration Possibilities

• ComfyUI pipelines – drag‑and‑drop nodes for prompt input, latent diffusion, and video decoding.
• Python API – load the .safetensors file with torch.load and run inference via the diffusers library (custom wrappers may be required).
• REST services – host the model behind a Flask/FastAPI endpoint for on‑demand video generation.