Wan2.1-VAE-upscale2x

Technical Overview

Model ID: spacepxl/Wan2.1-VAE-upscale2x
Model Name: Wan2.1‑VAE‑upscale2x
Author: spacepxl
Base Model: Wan‑AI/Wan2.1‑T2V‑14B

Wan2.1‑VAE‑upscale2x is a decoder‑only finetune of the original Wan2.1 VAE that embeds a 2× up‑scaling step directly inside the decoder. The encoder is left untouched, preserving the latent space of the original Wan2.1 model, while the final 3‑D convolution of the decoder is expanded from 3 output channels (RGB) to 12 channels. Those 12 channels are later rearranged with a pixel‑shuffle operation to produce a full‑resolution image at twice the spatial size of the latent representation.

Key features and capabilities

• Integrated 2× up‑scaling – no external super‑resolution model required.
• Significant reduction of the “Wan speckles”, “polka dots”, and grain artifacts that were common in the original decoder.
• Improved skin‑tone fidelity and hair detail, especially on real‑world photographs.
• Works out‑of‑the‑box with both Wan and Qwen pipelines because they share the same latent space.
• Free up‑scaling – the extra resolution comes at virtually the same compute cost as the original decoder.

Architecture highlights

• Encoder: frozen copy of the Wan2.1 encoder (unchanged).
• Decoder: original Wan2.1 decoder with the last Conv3d layer replaced (3 → 12 channels).
• Pixel‑shuffle (upscale factor = 2) applied after decoding to convert the 12‑channel tensor into a high‑resolution RGB image.
• Training loss combines L1, LPIPS, Frequency Distribution Loss (FDL) and a high‑weight PatchGAN discriminator to balance perceptual quality and sharpness.

Intended use cases

• High‑resolution image generation pipelines (e.g., “high‑res fix” in Stable Diffusion workflows).
• Photorealistic portrait, fashion, and product rendering where skin and hair detail matter.
• Any scenario where you want a clean, artifact‑free up‑scaled output without adding a separate super‑resolution model.

Benchmark Performance

Because the author focused on human perception rather than classic pixel‑wise metrics, the README does not list PSNR, SSIM, or LPIPS numbers. Instead, the model is evaluated visually via side‑by‑side comparisons:

• Original VAE vs. 2× VAE – shows the dramatic reduction of speckles.
• High‑res fix over ultrasharp upscaler vs. over 2× decoder – demonstrates that the integrated decoder produces sharper, more natural textures than a post‑hoc upscaler.

These visual benchmarks are crucial for generative image models because pixel‑wise scores often mis‑align with what users perceive as “clean” or “realistic”. Compared to the vanilla Wan2.1 VAE, the up‑scale2x version delivers:

• ~2× higher spatial resolution at the same decode cost.
• Noticeably fewer grain artifacts, especially on smooth skin and hair.
• Sharper edges and better texture fidelity, thanks to the high‑weight GAN loss.

Hardware Requirements

VRAM for inference – The decoder outputs a 12‑channel tensor at the latent resolution (e.g., 256 × 256 × 12). After pixel‑shuffle this becomes a 512 × 512 RGB image. In practice, a 6 GB GPU can run a single image batch comfortably; 8 GB+ is recommended for batch sizes > 1 or for use in ComfyUI pipelines.

Recommended GPU – Any modern NVIDIA GPU with at least 8 GB VRAM (e.g., RTX 3060, RTX 3070, RTX 3080, RTX 4090). The model was trained on a single RTX 5090, so inference is far less demanding.

CPU & RAM – A recent multi‑core CPU (e.g., AMD Ryzen 5 5600X or Intel i5‑12600K) is sufficient. System RAM of 16 GB is advisable to hold the model weights (≈ 2 GB) and intermediate tensors.

Storage – The model package (safetensors + config) is ~ 2 GB. SSD storage is recommended for fast loading; HDD works but will add noticeable latency on first load.

Performance characteristics – Decoding cost is comparable to the original Wan2.1 VAE (≈ 0.03 s per 512 × 512 image on an RTX 3090). The pixel‑shuffle step is negligible (< 0.005 s). Therefore, you gain double resolution without a proportional increase in inference time.

Use Cases

Primary applications

• High‑resolution portrait generation for fashion, cosmetics, and virtual try‑on.
• Product photography up‑scaling where clean edges and texture fidelity are essential.
• Creative workflows that rely on “high‑res fix” steps in Stable Diffusion or ComfyUI pipelines.

Real‑world examples

• A marketing agency uses the model to upscale AI‑generated model shots from 256 × 256 to 512 × 512, eliminating the grain that previously required a separate super‑resolution pass.
• Game asset creators generate concept art with realistic skin and hair, then feed the 2× decoder output directly into texture‑baking pipelines.
• Virtual avatar platforms employ the model to produce high‑quality facial renders for live‑stream avatars, where low latency and high visual fidelity are critical.

Integration possibilities

• ComfyUI – via the ComfyUI‑VAE‑Utils custom nodes.
• Diffusers – using AutoencoderKLWan as shown in the README.
• Direct PyTorch pipelines – the decoder can be called as a standard nn.Module and combined with any latent‑generation model that shares the Wan2.1 latent space.

Training Details

Methodology – The encoder from the pretrained Wan2.1 VAE is frozen to keep the latent space unchanged. The decoder’s final Conv3d layer is replaced to output 12 channels, enabling a pixel‑shuffle up‑scale. Training runs for 300 k steps on a single RTX 5090 (≈ 40 h). The batch size is 4, with a base resolution of 256 × 256 (up‑scaled to 512 × 512 by the decoder).

Loss composition

• L1 reconstruction loss – ensures basic pixel fidelity.
• LPIPS – regularizes perceptual similarity while being kept low to avoid speckle artifacts.
• FDL – encourages realistic texture statistics and local frequency distribution.
• PatchGAN discriminator with spectral norm – provides a high‑weight adversarial signal for sharpness.

Dataset – The model is trained “almost exclusively on real images”. No specific dataset name is given, but the author stresses that the data is licensed correctly (CC‑BY or similar). Consequently, the model excels on photographs but may struggle with anime, line‑art, or text‑heavy images.

Fine‑tuning capabilities – Because the encoder is frozen, you can fine‑tune the decoder on a domain‑specific dataset (e.g., anime) by swapping the loss weights or adding a domain‑specific discriminator. The same ComfyUI and Diffusers wrappers support custom latent normalization, making adaptation straightforward.

Related Papers

The README references two key research components:

• LPIPS – Learned Perceptual Image Patch Similarity – a perceptual loss that aligns with human judgment.
• Frequency Distribution Loss (FDL) – a VGG‑based perceptual loss that emphasizes realistic texture statistics.

Both papers underpin the loss formulation used to train the 2× decoder, balancing sharpness (GAN) with perceptual consistency (LPIPS, FDL). The model also builds on the architecture of the original Wan2.1 VAE, which itself draws from the latent‑space design of Stable Diffusion’s VAE.

Licensing Information

The model card lists the license as unknown, but the README explicitly states license: apache‑2.0. If the Apache‑2.0 license is indeed the governing terms, the following applies:

• Commercial use: Permitted without fee.
• Modification & redistribution: Allowed, provided you retain the original copyright notice and include a copy of the Apache‑2.0 license.
• Patent grant: The license includes an explicit patent license from contributors.
• Attribution: You must credit the original author (spacepxl) and the upstream Wan‑AI project.

If the true license remains truly “unknown”, you should treat the model as all‑rights‑reserved until clarification is received. In that case, commercial deployment would be risky without explicit permission from the author. Always verify the license on the Hugging Face model card before using it in a production environment.