Diffusion Models: Denoising, Guidance and the Text-to-Image Breakthrough

Diffusion models change the unit of image generation. Rather than ask a network to produce a finished picture in one decision, they ask it to make a sequence of local denoising decisions. That seemingly modest change underpins both DALL·E 2, whose paper names the architecture unCLIP, and Imagen.

These systems do not share one generic text-to-image recipe. unCLIP generates an intermediate CLIP image representation; Imagen conditions diffusion directly on a frozen language model. Their differences expose the choices that matter for deployment: which representation carries the prompt, how strongly guidance restricts the output, and how much sequential computation a request deserves.

Training teaches reversal, not image completion

A denoising diffusion probabilistic model starts with a fixed forward process. It adds small amounts of Gaussian noise to a training image over many timesteps until the original signal becomes statistically indistinguishable from noise. The process needs no learned parameters. A noise schedule controls how quickly the signal disappears.

The crucial property is that we can jump directly to any timestep. For an image x₀, sampled Gaussian noise ε and cumulative signal coefficient ᾱₜ, the corrupted image is:

xₜ = √(ᾱₜ)x₀ + √(1−ᾱₜ)ε

Training samples an image, a timestep and a noise realisation, then asks a neural network to predict the noise. Its simplified objective is the squared error between ε and εθ(xₜ, t). The timestep is supplied to the network, allowing shared weights to handle many noise levels. There is no need to run the whole chain for every training example: random noisy states provide independent training targets.

Generation reverses the direction. The sampler begins with fresh Gaussian noise and repeatedly uses the predicted noise to parameterise a distribution over the next, slightly cleaner state. In the original DDPM sampler, that transition also includes scheduled randomness. “Denoising” therefore does not mean subtracting an exactly known corruption once. It means following a learned probabilistic reverse chain, with early steps tending to establish broad structure and later steps refining detail.

This asymmetry is fundamental. Training can sample timesteps independently and parallelise the work. Inference must traverse an ordered trajectory because each state depends on the previous one. The quality of the final image is bought with serial network evaluations.

Guidance trades coverage for compliance

Conditioning tells the denoiser what to generate, but a conditional prediction alone does not force every output to honour the prompt. Guidance pushes each reverse step more strongly towards the condition.

Classifier guidance uses a separate classifier trained on noisy images. At every step, the gradient of the classifier’s log probability for the desired label shifts the reverse-process mean. Increasing that gradient’s scale improves class consistency and fidelity, but Dhariwal and Nichol measured a corresponding loss of recall: the model covers less of the data distribution. The best setting is therefore not the largest possible guidance weight.

Classifier-free guidance removes the separate classifier. During training, the model sometimes receives an empty condition instead of the caption or label. It consequently learns both conditional and unconditional predictions. At sampling time, the noise estimate becomes:

εguided = εuncond + s(εcond − εuncond)

At s = 1, this is the ordinary conditional estimate. Larger values extrapolate away from the unconditional prediction. The same network now supplies its own steering signal, which works for free-form text as well as class labels. GLIDE’s human evaluations favoured this approach over CLIP guidance for photorealism and caption similarity. They also exposed a measurement hazard: stronger CLIP guidance could improve a CLIP similarity score without producing corresponding human preference.

The cost appears in two places. Stronger guidance can reduce semantic diversity and amplify biases present in the data. A direct implementation also evaluates the denoiser twice per step, once with the prompt and once without it. Batching those evaluations can improve hardware utilisation, but it does not abolish the computation.

Two architectures for language-conditioned denoising

unCLIP samples a semantic plan

CLIP maps text and images into a shared embedding space. unCLIP turns generation into two problems. First, a prior converts a caption into a plausible CLIP image embedding, aided by the frozen CLIP text embedding. The authors tested autoregressive and diffusion priors; their diffusion prior performed better at comparable model size with less training compute. Second, a diffusion decoder approximately inverts the CLIP image encoder, producing pixels from that image embedding and, optionally, the caption.

A 64×64 decoder establishes the image, then two diffusion upsamplers produce 256×256 and 1024×1024 outputs. The image embedding acts like a semantic plan rather than a compressed copy of the final pixels. Many images can match it, and the decoder supplies the missing detail. This helps explain the diversity of unCLIP’s variations. When the sampled embedding is held fixed, stronger decoder guidance can improve visual fidelity without forcing the scene’s semantics to converge as sharply as GLIDE’s.

The representation also creates a bottleneck. The paper reports weaker binding between objects and attributes, unreliable rendered text and lost detail in complex scenes. A CLIP vector need not encode which cube should be red or preserve spelling precisely. No decoder can reliably recover information that the intermediate representation has discarded. The prior adds another learned component and another sampling process, but it makes that semantic bottleneck explicit and generative.

Imagen keeps the language sequence

Imagen removes both the CLIP prior and the intermediate image embedding. A frozen T5-XXL encoder, pretrained on text alone, maps the prompt to a sequence of embeddings. Cross-attention feeds those embeddings to a 64×64 text-conditional diffusion model and then to two text-conditional super-resolution models, following the same progression to 1024×1024.

This is a different bet on representation. Instead of compressing the prompt into a shared visual-semantic vector, Imagen retains token-level language features throughout the cascade. Its authors found that scaling the frozen text encoder improved fidelity and alignment more than scaling the image U-Net in their experiments. That result argues for investing in language understanding, not for treating text encoder size as a universal law.

Freezing the encoder has a practical benefit: training embeddings can be computed offline, keeping the large language model’s activations and gradients out of diffusion training. Direct conditioning also avoids unCLIP’s learned prior. In return, the system carries a large text encoder into the serving stack and inherits limitations from text-only pretraining as well as image-text data.

Imagen also shows why strong guidance needs numerical control. Large weights can push the predicted clean image outside the [-1, 1] range seen during training. Because each sampler iteration consumes its own previous output, those errors can produce saturation or divergence. Static clipping prevents some failures but leaves oversaturated images. Dynamic thresholding instead clips at a high percentile of the predicted pixel magnitudes and rescales when that threshold exceeds one, making large guidance weights more usable.

Comparison of unCLIP sampling a CLIP image embedding before its decoder and Imagen feeding frozen T5-XXL embeddings to three diffusion stages. — Both systems iteratively denoise images. unCLIP first samples a CLIP image embedding; Imagen conditions every stage of its resolution cascade directly on frozen T5-XXL text embeddings.

The inference bill is an architectural feature

The original DDPM used a thousand network evaluations to generate an image. Cascades add further iterative samplers, and classifier-free guidance increases work within each step. Higher resolution therefore brings more than a larger output tensor: it can multiply latency, accelerator occupancy and the cost of offering several candidates per prompt.

Denoising diffusion implicit models address the trajectory rather than the denoising objective. DDIM defines a family of non-Markovian forward processes with the same training objective as DDPM. A pretrained denoiser can consequently use a deterministic, shortened sampling trajectory without retraining. Fewer steps still trade computation against sample quality, but the sampler becomes an explicit control over that trade-off instead of a fixed consequence of training.

A service can make similar choices across the pipeline. Cache text embeddings when prompts repeat. Batch conditional and unconditional passes where memory permits. Generate a low-resolution candidate before paying for super-resolution. Treat guidance, step count and candidate count as budgeted controls, then measure their effect on the actual prompt distribution rather than assuming that more computation always helps.

Evaluation needs the same discipline. FID summarises distributional fidelity and diversity, while CLIP scores estimate prompt alignment through a particular representation. Neither establishes compositionality or human preference on its own. Human review remains necessary, especially for unusual combinations, people and rendered text. Dataset audits and access controls belong in the same assessment: GLIDE found that guidance could amplify existing biases, and Imagen’s authors withheld code and a public demo because of data and misuse concerns.

The text-to-image advance is thus not a single architecture or a free gain in quality. Denoising supplies a tractable training objective; guidance narrows the output towards a prompt; the conditioning design determines what language information survives. unCLIP makes a semantic latent explicit and samples it. Imagen preserves a language sequence and spends compute on a direct conditional cascade. Engineering a useful system means choosing those compromises deliberately, then accounting for every sequential step.