What is a diffusion model?

A generative model trained to undo noise. Training adds noise to real images a little at a time, over 1,000 steps in the original DDPM paper, and teaches a network to predict the noise that was added. Generating starts from pure noise and runs the steps backwards, one prediction at a time.

What are steps in Stable Diffusion?

How many times the sampler removes a slice of noise. More steps follow the score more closely; fewer are faster but rougher. On the stage, 50 DDPM steps leave 95% of the dots on the picture and 10 steps leave 70%. Samplers such as DDIM were designed to get good images from fewer steps.

Why does the same prompt give different images?

Each image starts from different random noise, and the sampler carries that noise to a different picture. Fix the seed and the same noise and settings give the same image again. On the stage, change the seed: the smiley comes back with every dot in a new place.

What is the score in diffusion?

The direction, at a given noise level, in which an image would become more likely: the gradient of the log of the noised images' density. Sampling follows it downhill in noise. On the stage it pulls each dot toward the blobs it most likely came from, weighted by how likely each one is.

Live model · AI image generators

How AI Image Generators Actually Work

Diffusion image generators start from pure random noise and remove it in small steps. At each step a trained network looks at the noisy image and predicts the noise in it, which points toward a likelier image; the sampler moves a little that way. After dozens of steps (a thousand in the original method) the noise is a picture.

From noise to a picture in 50 steps

400 dots of pure noise. Each will become one point of a picture: a smiley made of 25 blurry blobs, dashed. DDPM moves each dot a little way along the score, then adds a little fresh noise, less each step.

StopDDPM, 50 steps

StepsSamplerNoise scheduleSeed

CH1 Ends on an eye
CH2 Ends on the smile
CH3 Ends on the outline

How to read the stage

Each dot is one “image”: a point in two dimensions, so the picture is where dots pile up, a smiley made of 25 blurry blobs. All 400 dots start as pure noise. Each of 50 DDPM steps moves every dot along the score, the direction in which the noised picture's density rises fastest, then adds a little fresh noise. The gray image shows where the finished picture would be, guessed from each dot's position.

The layout comes first, the details last

For the first 24 steps the noise level stays above 0.97, and every guess sits in one smudge. At step 27, 3% of the guesses sit on a stroke; by step 42, 52% do. The last steps only sharpen. At the end 95% of the dots sit on the picture, and 81 went to the eyes, 93 to the smile and 226 to the outline. The eyes carry 6 of the picture's 29 parts of weight, about 83 dots' worth.

Training runs the other way

Training takes real images and buries them in noise: after t of 1,000 steps, a dot is √ᾱ of itself plus √(1 − ᾱ) of noise. With Ho and colleagues' linear schedule, the signal is 0.28 at step 500 and 0.08 by step 700, so the last 300 steps are nearly pure noise. Nichol and Dhariwal's cosine schedule keeps 0.70 at step 500, and on the stage above it decides the layout by step 24 of 50, not 42.

Burying the picture in noise

400 dots drawn from the picture: each picks one of its 25 blobs, the heavy eyes more often, and lands somewhere inside it. These are the training images.

StopForward process, linear

Noise scheduleSeed

CH1 From an eye
CH2 From the smile
CH3 From the outline

What the network is for

A real model never knows the density of all pictures. It's trained to look at a noised image and predict the noise that was added, and that prediction, rescaled, is the score; Song and colleagues showed sampling needs nothing else. This sampler computes the score exactly, because the smiley's density is a known formula. Real models learn it with a network over every pixel: a 512 by 512 color image is 786,432 numbers.

What the model leaves out

Real generators learn the score from huge collections of pictures. Stable Diffusion runs the whole process on a compressed 64 × 64 × 4 version of a 512 × 512 image and decodes it at the end, and a text prompt steers the network through cross-attention. The DDIM paper reports high-quality images 10 to 50 times faster than DDPM. In this tiny model, 10 steps leave 70% of DDPM's dots on the picture and 63% of DDIM's.

Questions

What is a diffusion model?: A generative model trained to undo noise. Training adds noise to real images a little at a time, over 1,000 steps in the original DDPM paper, and teaches a network to predict the noise that was added. Generating starts from pure noise and runs the steps backwards, one prediction at a time.
What are steps in Stable Diffusion?: How many times the sampler removes a slice of noise. More steps follow the score more closely; fewer are faster but rougher. On the stage, 50 DDPM steps leave 95% of the dots on the picture and 10 steps leave 70%. Samplers such as DDIM were designed to get good images from fewer steps.
Why does the same prompt give different images?: Each image starts from different random noise, and the sampler carries that noise to a different picture. Fix the seed and the same noise and settings give the same image again. On the stage, change the seed: the smiley comes back with every dot in a new place.
What is the score in diffusion?: The direction, at a given noise level, in which an image would become more likely: the gradient of the log of the noised images' density. Sampling follows it downhill in noise. On the stage it pulls each dot toward the blobs it most likely came from, weighted by how likely each one is.