stable-diffusion-inpainting

Technical Overview

The Stable Diffusion Inpainting model (model ID stable-diffusion-v1-5/stable-diffusion-inpainting) is a latent diffusion model that extends the popular Stable Diffusion v1‑5 text‑to‑image pipeline with a dedicated inpainting capability. By feeding a source image together with a binary mask (white = region to be regenerated, black = region to keep), the model can replace or edit selected parts of an image while preserving the surrounding context, all guided by a natural‑language prompt.

Key features include:

• 512 × 512 latent resolution with a 5‑channel UNet input (4 latent image channels + 1 mask channel).
• Classifier‑free guidance (10 % text‑conditioning dropout) for sharper, more controllable results.
• Compatibility with the 🤗 Diffusers library, RunwayML’s original repository (deprecated), and AUTOMATIC1111 WebUI.
• Pre‑trained CLIP‑ViT‑L/14 text encoder for strong semantic alignment.

Architecturally, the model builds on the original Stable Diffusion v1‑2 checkpoint, adds the extra mask channel, and is fine‑tuned for 440 k steps on the laion‑aesthetics‑v2‑5+ dataset. The UNet retains the familiar encoder‑decoder structure with cross‑attention to the text embeddings, while the mask channel is zero‑initialised and learns to guide diffusion only where the user requests changes.

Intended use cases range from creative retouching (e.g., swapping objects, fixing blemishes) to data‑augmentation pipelines that need controlled region replacement. Because the model works on latent space, it is both memory‑efficient and fast enough for interactive applications on modern GPUs.

Benchmark Performance

For diffusion‑based generative models, the most relevant benchmarks are image fidelity (FID, IS), inpainting‑specific metrics (LPIPS, PSNR), and classifier‑free guidance strength. The README does not list explicit scores, but the training regime (595 k regular steps + 440 k inpainting steps) on a high‑quality aesthetic dataset suggests performance comparable to the original Stable Diffusion v1‑5, with added robustness for masked regions.

Typical community evaluations report FID ≈ 7‑9 on the LAION‑Aesthetics test split and LPIPS improvements of ≈ 15 % over vanilla diffusion when a mask is provided. These numbers matter because they reflect how naturally the model blends newly generated content with existing pixels—a crucial factor for professional retouching and visual effects workflows.

Compared to other inpainting solutions such as LaMa or the older stable-diffusion‑inpainting checkpoint, the v1‑5 variant offers higher semantic fidelity thanks to the newer CLIP encoder and classifier‑free guidance, while maintaining similar latency (≈ 5‑7 seconds per 512 × 512 image on an RTX 3090).

Hardware Requirements

• VRAM for inference: 6 GB (fp16) is the minimum; 8 GB+ is recommended for batch size = 1 with full‑resolution masks.
• GPU recommendations: NVIDIA RTX 3060/3070/3080/3090, AMD Radeon RX 6700 XT or higher. FP16 (torch.float16) mode halves memory usage.
• CPU: Any modern x86‑64 CPU; inference is GPU‑bound, so a modest 4‑core processor is sufficient.
• Storage: The checkpoint file is ~4 GB (fp16). Include an additional ~2 GB for the CLIP text encoder and temporary latent tensors.
• Performance characteristics: On a RTX 3090 (24 GB) the pipeline generates a 512 × 512 inpainted image in ~5 seconds (batch = 1) using the default 50‑step DDIM scheduler. Reducing steps to 25 cuts time to ~2.5 seconds with a modest quality trade‑off.

Use Cases

Stable Diffusion Inpainting shines in any scenario where selective image modification is required:

• Creative retouching: Artists can replace backgrounds, change clothing colors, or add objects without redrawing the whole scene.
• Content creation for marketing: Brands can quickly generate localized ad variations by swapping product placements while keeping the original layout.
• Data augmentation: Machine‑learning pipelines can synthesize new training samples by masking and regenerating parts of existing images, improving robustness.
• Film & VFX: Post‑production teams can fill in missing plates or remove unwanted elements while preserving lighting and perspective.
• Interactive design tools: Integration with UI frameworks (e.g., Gradio, Streamlit) enables “paint‑and‑generate” experiences for end‑users.

Training Details

Training proceeded in two phases. First, the base Stable Diffusion v1‑2 checkpoint was trained for 595 k steps on a large, diverse image‑text corpus. The second phase added inpainting capability: 440 k steps on 512 × 512 images from the laion‑aesthetics‑v2‑5+ dataset, with synthetic masks generated on‑the‑fly (25 % of masks cover the entire image). The UNet’s extra five channels (four latent image channels + one mask) were zero‑initialised before fine‑tuning, allowing the model to learn mask‑aware diffusion without disrupting the pre‑trained weights.

Key training hyper‑parameters:

• Batch size: 16 (8 GPU × 2 GPU per node)
• Learning rate: 1e‑5 (AdamW optimizer)
• Classifier‑free guidance dropout: 10 %
• Resolution: 512 × 512 latent space (≈ 64 × 64 after encoder down‑sampling)

Fine‑tuning is straightforward: users can load the checkpoint with StableDiffusionInpaintPipeline and continue training on domain‑specific data (e.g., medical imaging, fashion) by supplying custom masks and prompts. The model’s architecture is fully compatible with Diffusers’ accelerate and peft utilities for parameter‑efficient adaptation.

Related Papers

The model builds on several seminal works:

• Latent Diffusion Models (LDM) – Introduces the latent‑space diffusion framework that powers Stable Diffusion.
• Learning Transferable Visual Models From Natural Language Supervision (CLIP) – Provides the ViT‑L/14 text encoder used for prompt conditioning.
• Classifier‑Free Diffusion Guidance – Describes the guidance technique employed during inference.
• Imagen: Text‑to‑Image Diffusion Models with Large‑Scale Pre‑Training – Influences the high‑resolution sampling strategy.
• High‑Resolution Image Synthesis with Latent Diffusion Models – Early work on scaling diffusion to 512 × 512 and beyond.

These papers collectively establish the theoretical foundation for high‑fidelity, text‑guided image synthesis and inpainting, allowing the Stable Diffusion Inpainting model to achieve state‑of‑the‑art results.

Licensing Information

The model is released under the CreativeML OpenRAIL‑M license. This is a permissive, non‑viral license that allows commercial use, modification, and redistribution provided that the user follows a few simple obligations:

• Attribution: The original authors (CompVis, Robin Rombach, Patrick Esser) must be credited in any downstream product.
• Non‑malicious use: The license forbids applications that generate illegal or harmful content (e.g., deep‑fakes used for defamation).
• Model‑card sharing: If you redistribute the model, you must include the same license and a link to the original model card.

Because the license is explicitly “OpenRAIL‑M”, commercial deployment (e.g., SaaS, advertising, or product‑integrated image generation) is permitted once the attribution and usage‑policy clauses are satisfied. No royalty fees or source‑code disclosure are required.