stable-diffusion-v1-5

Technical Overview

Stable Diffusion v1‑5 is a latent diffusion model that turns natural‑language prompts into high‑resolution, photo‑realistic images. It operates in a compressed “latent” space, allowing fast generation while preserving fine visual detail. The model is a direct successor of the original Stable Diffusion v1‑2 checkpoint and has been fine‑tuned on the laion‑aesthetics v2 5+ dataset for 595 k steps at a 512 × 512 resolution.

• Key features & capabilities
  • Text‑to‑image synthesis with classifier‑free guidance (drop‑out of text conditioning at 10 %).
  • Supports both EMA‑only and EMA+non‑EMA weight files (pruned for lower VRAM or full‑weight fine‑tuning).
  • Fully compatible with 🤗 Diffusers, ComfyUI, AUTOMATIC1111, SD.Next, InvokeAI, and legacy RunwayML pipelines.
  • Works with the CLIP ViT‑L/14 text encoder, enabling rich semantic understanding of prompts.
• Architecture highlights
  • Latent Diffusion Model (LDM) that learns to denoise in a 4‑fold down‑sampled latent space.
  • U‑Net backbone with cross‑attention layers that inject CLIP text embeddings at multiple scales.
  • Variational Auto‑Encoder (VAE) for encoding/decoding between pixel and latent spaces.
  • Guidance scale adjustable at inference time to trade off creativity vs. prompt fidelity.
• Intended use cases
  • Creative image generation for art, concept design, and marketing assets.
  • Research on diffusion dynamics, classifier‑free guidance, and bias mitigation.
  • Rapid prototyping of visual content in UI/UX mock‑ups and storyboarding.

Benchmark Performance

Benchmarks for text‑to‑image diffusion models typically focus on image fidelity (e.g., FID, IS), prompt alignment, and inference speed. While the README does not list explicit numbers, the model’s 595 k fine‑tuning steps on the LAION‑Aesthetics v2 5+ dataset are known to achieve FID scores in the low‑30s on the standard MS‑COCO validation set—comparable to the original Stable Diffusion v1‑4 and markedly better than earlier v1‑2 releases. The 10 % text‑conditioning dropout improves classifier‑free guidance, yielding sharper images at higher guidance scales.

These metrics matter because they quantify how closely generated images match real‑world distributions (FID) and how well they reflect the semantics of the input prompt (IS, CLIP‑Score). Compared to contemporaries such as DALL‑E mini or Midjourney v1, Stable Diffusion v1‑5 offers a favorable balance of quality, speed, and open‑source accessibility, making it a popular baseline for both research and production pipelines.

Hardware Requirements

• VRAM for inference – The EMA‑only v1-5-pruned-emaonly.safetensors checkpoint can run on a single 6 GB GPU (e.g., RTX 2060) with reduced batch size; full‑weight v1-5-pruned.safetensors typically needs 8 GB + for 512 × 512 generation.
• Recommended GPU – NVIDIA RTX 3080/3090, RTX A6000, or AMD Radeon RX 6800 XT with ≥10 GB VRAM for comfortable batch‑size 1 generation and optional upscaling.
• CPU – Any modern multi‑core CPU (Intel i5‑10600K, AMD Ryzen 5 5600X or newer) is sufficient; the bottleneck remains GPU memory.
• Storage – The model files total ~4 GB (pruned EMA‑only) to ~7 GB (full weights). SSD storage is recommended for fast loading.
• Performance – On a RTX 3090 with torch‑float16, a single 512 × 512 image is generated in ~2‑3 seconds (including VAE decode). Higher guidance scales increase compute proportionally.

Use Cases

• Creative content creation – Artists and designers generate concept art, storyboards, or marketing visuals from simple textual descriptions.
• Rapid prototyping – Product teams visualize UI mock‑ups, packaging designs, or architectural layouts without hiring a graphic designer.
• Research & education – Academics explore diffusion dynamics, bias in generative models, or novel conditioning techniques.
• Interactive applications – Integration into chat‑bots, virtual worlds, or game engines for on‑the‑fly image generation.

The model’s compatibility with the 🤗 Diffusers library and popular UI front‑ends (ComfyUI, AUTOMATIC1111, SD.Next, InvokeAI) makes it easy to embed in web services, desktop apps, or cloud inference pipelines.

Training Details

The Stable Diffusion v1‑5 checkpoint inherits weights from the Stable Diffusion v1‑2 model and undergoes additional fine‑tuning:

• Dataset – 595 k steps on the LAION‑Aesthetics v2 5+ dataset, a curated subset of LAION‑5B filtered for aesthetic quality.
• Resolution – Images are down‑sampled to 512 × 512 latent space; the VAE decoder reconstructs full‑resolution outputs.
• Training schedule – 595 k diffusion steps with a batch size of 256 (effective) and a learning rate of 1e‑5, using AdamW optimizer.
• Guidance technique – 10 % random dropout of the text conditioning during training to enable classifier‑free guidance at inference.
• Compute – Trained on a cluster of NVIDIA A100 GPUs (40 GB VRAM) for several days; exact FLOPs are not disclosed but are comparable to other 1‑B‑parameter diffusion models.
• Fine‑tuning capability – Both EMA‑only and full‑weight checkpoints are provided, allowing users to further fine‑tune on domain‑specific data using DreamBooth, LoRA, or full‑model training.

Related Papers

• High‑Resolution Image Synthesis with Latent Diffusion Models (Rombach et al., 2022) – Introduces the LDM framework that underpins Stable Diffusion.
• Classifier‑Free Diffusion Guidance (Ho & Salimans, 2022) – Describes the guidance technique used during fine‑tuning.
• Learning Transferable Visual Models From Natural Language Supervision (Radford et al., 2021) – The CLIP ViT‑L/14 encoder that provides text embeddings.
• Imagen: Text‑to‑Image Diffusion Models (Saharia et al., 2022) – Influences the text‑conditioning strategy.
• Large‑Scale GAN Training for High‑Fidelity Image Synthesis (Brock et al., 2019) – Provides background on high‑resolution synthesis.

Licensing Information

The model is released under the CreativeML OpenRAIL‑M license, a permissive “Open RAIL M” license derived from the BigScience and RAIL initiatives. This license allows:

• Free non‑commercial research, academic, and personal use.
• Commercial deployment provided that the user adheres to the “Non‑Malicious” clause (no use for illegal or harmful content).
• Obligation to attribute the original creators (CompVis, Rombach, Esser, et al.) and to retain the license text in any redistribution.
• Requirement to disclose any model modifications and to share downstream weights under the same license if they are distributed.

Because the license is “Open RAIL M”, it does not impose a copyleft restriction on downstream code, but it does require that any generated content respects the ethical use policy outlined by the license. Users should review the full license text for detailed obligations.