Why do AI videos flicker or morph between frames?

Each frame is denoised from its own noise, and only attention across frames makes them agree. When that attention is weak, or a long video is built from pieces made separately, frames settle on slightly different scenes and things change shape from one frame to the next. Here, frames denoised alone switch version 8 times in 12 frames.

How is AI video generation different from AI image generation?

The denoising is the same idea; a video model denoises a stack of frames at once and adds attention along time, so each frame sees the others. It also has to keep the scene consistent and the motion smooth, which is why video models are larger and slower than image ones.

What does the guidance scale do?

It sets how hard the model pushes its guess toward the prompt and away from its guess with no prompt. Low guidance follows the prompt loosely and varies more; high guidance follows it closely, until colors saturate and details burn out, as at guidance 8 here.

What does the seed do in AI video?

It sets the starting noise. The same prompt and seed give the same clip; a new seed gives different noise, which the model may read as a different version of the prompt. Here seed 7 lands on version A and seed 4 on version B.

How can Sora make a video a minute long?

OpenAI describes Sora as a diffusion transformer that denoises spacetime patches of a compressed video, so attention spans the whole clip at once. Reviewers note it keeps objects persistent over long shots, and still struggles with the timing of events.

Live model · AI video generation

How AI Video Generation Actually Works

A video model starts from pure noise for every frame and removes it in steps, the way an image model does, but it denoises all the frames together. Temporal attention lets each frame look at the others while it's denoised, so they agree on one scene. Frames denoised apart disagree, and the clip flickers.

Twelve frames of noise, denoised together

The prompt is “a ball rolls left to right”. The model starts from 12 frames of pure noise, one sample per pixel, and will denoise them together in 12 steps. In this model every frame attends to all 12.

StopVideo diffusion, guidance 1

PromptGuidanceFrames attend toSeed

CH1 Version A
CH2 Version B
CH3 Version C
CH4 Version D

Watch the Python behind this model run, line by line

How to read the stage

The strip is the whole clip, 12 frames of 24 × 14 pixels, after each of 12 denoising steps; the screen above shows one frame large. The row under it is which of four versions of “a ball rolls left to right” each frame settled on. At step 1 all 12 frames pick version A, and they keep it: 0 switches. The finished clip sits 1 gray level from version A. Then it plays, frame by frame.

Every frame at once

Ho et al.'s video diffusion model took an image denoiser and added a temporal attention block after each spatial one, attending across frames, so the network denoises a block of frames as one. Google's Lumiere generates a clip's whole duration in one pass for the same reason: models that make distant keyframes first and fill in between found global consistency hard. Here, attending to all 12 frames means each frame's choice is made with the others in view.

Why frames flicker

Same seed, same noise, three ways. Attending to the whole clip: 0 switches, and pixels change 7 gray levels a frame, the ball's own motion. Neighbors only: 4 switches, as agreement drifts down the clip. Each frame alone: 8 switches and 15 gray levels a frame, the floor jumping as frames disagree. Ho et al. saw the same when they extended videos block by block the naive way: blocks looked like uncorrelated samples.

One seed, three ways of attending

One prompt, “a ball rolls left to right”, one seed, so the same starting noise, sampled three times. The only change is which frames each frame looks at while it's denoised. Frame 1 of each clip is on its screen; step to play them.

StopTemporal attention

PromptSeed

Frame1 of 12Switches0 · 4 · 8

Whole clip

1 of 12

Neighbors only

1 of 12

Each frame alone

1 of 12

Whole clip: versions

Neighbors only: versions

Each frame alone: versions

How much each frame changes from the one before

Whole clip
Neighbors only
Each frame alone

The three clips

Attends to	Switches	Change
All 12	0	7
Neighbors	4	10
Itself	8	15

Change: gray levels a pixel moves per frame, on average.

CH1 Version A
CH2 Version B
CH3 Version C
CH4 Version D

What guidance trades

Classifier-free guidance takes the guess with the prompt and pushes it away from the guess without one. Here, guidance 0 gives a ghost of every clip the model knows, 30 gray levels from the prompt. Guidance 1 is the prompt's guess alone, 1 level off. At 3, 24% of pixels are pushed past black or white and clipped; at 8, 88%. Imagen's authors describe the same saturation at high guidance weights.

One seed, five guidance strengths

“a ball rolls left to right”, one seed, sampled five times at guidance 0, 1, 1.5, 3 and 8. Guidance takes the model's guess with the prompt and pushes it further from its guess without one: x̂ = x̂u + w·(x̂c − x̂u).

StopGuidance 0 to 8

PromptSeed

What the model leaves out

The twelve clips are drawn by code, not learned. A real model learns its guesses with a network trained on millions of videos, attends softly rather than picking the closest version, and works on compressed latent patches rather than pixels, as Sora does. Its guess with a prompt is vague, so guidance above 1 sharpens it; here that guess is exact, so you see only guidance's cost. Frames are 24 × 14 gray pixels, not HD.

Questions

Why do AI videos flicker or morph between frames?: Each frame is denoised from its own noise, and only attention across frames makes them agree. When that attention is weak, or a long video is built from pieces made separately, frames settle on slightly different scenes and things change shape from one frame to the next. Here, frames denoised alone switch version 8 times in 12 frames.
How is AI video generation different from AI image generation?: The denoising is the same idea; a video model denoises a stack of frames at once and adds attention along time, so each frame sees the others. It also has to keep the scene consistent and the motion smooth, which is why video models are larger and slower than image ones.
What does the guidance scale do?: It sets how hard the model pushes its guess toward the prompt and away from its guess with no prompt. Low guidance follows the prompt loosely and varies more; high guidance follows it closely, until colors saturate and details burn out, as at guidance 8 here.
What does the seed do in AI video?: It sets the starting noise. The same prompt and seed give the same clip; a new seed gives different noise, which the model may read as a different version of the prompt. Here seed 7 lands on version A and seed 4 on version B.
How can Sora make a video a minute long?: OpenAI describes Sora as a diffusion transformer that denoises spacetime patches of a compressed video, so attention spans the whole clip at once. Reviewers note it keeps objects persistent over long shots, and still struggles with the timing of events.