Diffusion and Flow Distillation

§0 TL;DR Cheat Sheet

💡 Diffusion / Flow Distillation in 9 sentences — compress a 50-1000 NFE teacher into a 1-4 NFE student. One-page interview essentials (full derivations in §1-§9).

1. Why: diffusion sampling defaults to 50-1000 NFE, with network forward passes accounting for >95% of total latency; the production-deployment threshold is typically ≤ 4 steps (real-time chat / mobile / video generation). This tutorial covers only few-step / one-step distillation, not RL post-training.

2. Trade-off: fewer steps usually hurt quality — naive uniform-skip DDIM at 4 steps is barely usable. The essence of distillation is using the teacher's 50-step trajectory/distribution as supervision to train a student that jumps in one step.

3. Three major technical routes: (a) trajectory matching (progressive distillation / CM / iCT / sCM / CTM / LCM / TCD) — make the student reproduce the teacher's ODE solution; (b) distribution matching (DMD / DMD2 / rCM) — use the score gap as the KL gradient and match two distributions; (c) adversarial (ADD / LADD / SDXL-Lightning / FLUX-schnell) — GAN loss + teacher distillation.

4. Consistency Models (Song 2023 ICML): learn a consistency function $f_\theta(x_t, t) \to x_0$, mapping any $x_t$ to the same $x_0$; the boundary f_\theta(x_{\sigma_\min}, \sigma_\min) = x_{\sigma_\min} is enforced by EDM-style preconditioning; CD (distillation, with teacher) / CT (training, no teacher).

5. iCT (Song-Dhariwal 2023): remove EMA target + pseudo-Huber loss (replacing LPIPS) + lognormal noise schedule + step-count curriculum, bringing CT close to CD quality.

6. sCM / TrigFlow (Lu-Song 2024 OpenAI): continuous-time CM with $x_t = \cos(t) x_0 + \sin(t) z$ (trig parametrization makes EDM precond + PF-ODE + CM same-form); 1.5B ImageNet 512 2-step FID 1.88, within 10% of the best diffusion.

7. DMD (Yin 2024 CVPR): treat student output as a "fake distribution", then use the fake score $s_\text{fake}$ minus the real score $s_\text{real}$ as the reverse-KL gradient that pushes the student: $\nabla_\theta \text{KL}(p_\text{fake} \| p_\text{real}) = \mathbb{E}[(s_\text{fake} - s_\text{real}) \cdot \partial G_\theta / \partial \theta]$. DMD2 (Yin 2024 NeurIPS) drops the regression loss, adds GAN, and supports multi-step students.

8. ADD / LADD (Sauer et al. 2023/2024 Stability): teacher score distillation + DINOv2 / VAE feature discriminator dual supervision. SDXL-Turbo 1-step 1024, SD3-Turbo 4-step; FLUX.1-schnell also in the LADD family.

9. LCM-LoRA (Luo 2023): package Latent CM training as a LoRA adapter; ~30 A100·h is enough to let any SD 1.5 / SDXL fine-tune generate in 4 steps without swapping the base model. The key enabler for production ecosystem.

§1 Intuition & why distillation is needed

1.1　Sampling cost is diffusion's Achilles' heel

Model / sampler	Typical NFE	1024² image latency (A100 fp16)
DDPM ancestral (1000 step)	1000	~90 s
DDIM (50 step)	50	~5 s
DPM-Solver++ (20 step)	20	~2 s
EDM Heun (35 step)	~35	~3.5 s
LCM (4 step)	4	~0.4 s
SDXL-Turbo (1 step)	1	~0.1 s
DMD2 / FLUX-schnell (1-4 step)	1-4	0.1-0.4 s

Production typically requires < 0.5 s (real-time chat) or < 1 s (mobile), and native diffusion is far over budget. Distillation is not an "optional optimization" — it's a necessary step to deploy diffusion.

1.2　Why naive few-step doesn't work

Change 50-step DDIM to 4-step uniform DDIM: the sampler's $\Delta t$ grows large, and first-order Euler error $O(\Delta t)$ blows up sharply, with high-frequency details collapsing and visible noise residue. Even with EDM Heun 2nd-order, 4-step typically has FID > 15 and is far from usable. Root cause: the teacher's ODE trajectory is curved (VP/VE path), and 4 steps can only approximate it with a coarse polyline.

💡 Core distillation idea — not changing the sampler, not lowering precision — but retraining a student that learns to "jump directly from any $x_t$ to $x_0$" (CM view) or "match the teacher's output distribution" (DMD view) or "produce images that fool the discriminator" (ADD view). The three views correspond to the three major schools.

1.3　Distillation vs accelerated samplers: fundamental difference

	Accelerated samplers (DDIM / DPM-Solver / Heun)	Distillation (CM / DMD / ADD)
Change training?	❌	✅ requires a new round of training
Change network?	❌ (same $\epsilon_\theta$)	✅ separate student network (or fine-tune from teacher)
NFE limit	10-20 (ODE solving accuracy limit)	1-4
Failure modes	discretization error	mode collapse / saturated colors / lack of diversity

Complementary relationship: production pipelines typically follow "first pick the sampler type → then distill" — e.g., SD3 uses RF (Euler-friendly) + LADD to distill to 4 steps; FLUX uses RF + LADD-schnell to distill to 1-4 steps.

1.4　Convention used throughout

Symbol	Meaning
$x_0$	clean data
$x_t$ ($t \in [0, T]$ or \sigma \in [\sigma_\min, \sigma_\max])	noised sample
$z, \epsilon$	$\mathcal{N}(0, I)$ noise
$\theta$ / $\phi$	student params / teacher params
$f_\theta(x_t, t)$	CM consistency function (CM output)
$G_\theta(z, t)$	one-step / few-step student generator
$s_\theta(x, t) \approx \nabla \log p_t(x)$	score
$D_\psi$	discriminator (used by ADD/LADD)
NFE	Number of Function Evaluations

⚠️ Time-direction pitfall (disambiguate up front) — the CM series uses EDM $\sigma$-time (\sigma_\min = 0.002, \sigma_\max = 80), DDPM uses $t \in [0, T]$, FM uses $t \in [0, 1]$. This tutorial standardizes per section: §2 CM/iCT/sCM use $\sigma$-time; §3 DMD uses $t \in [0, T]$; §4 ADD/LADD use $\sigma$-time (following EDM); §5 Flow family uses $t \in [0, 1]$ ($t=0$ noise / $t=1$ data).

§2 Consistency Models family

2.1　Consistency Models (CM, Song et al. 2023 ICML, arXiv:2303.01469)

Core definition: consistency function f: (x_t, t) \mapsto x_{\sigma_\min} is self-consistent along PF-ODE trajectories —

\boxed{\;f_\theta(x_t, t) = f_\theta(x_{t'}, t')\quad\text{for any } t, t' \in [\sigma_\min, \sigma_\max] \text{ on the same ODE trajectory}\;}

This yields one-step generation: x_0 \approx f_\theta(z \cdot \sigma_\max, \sigma_\max) with $z \sim \mathcal{N}(0, I)$.

Boundary condition: requires f_\theta(x, \sigma_\min) = x (identity at lowest noise) — enforced via EDM-style preconditioning:

$f_\theta(x, \sigma) = c_\text{skip}(\sigma)\, x + c_\text{out}(\sigma)\, F_\theta(x, \sigma)$

with c_\text{skip}(\sigma_\min) = 1, c_\text{out}(\sigma_\min) = 0. Song 2023's specific values (same form as EDM Karras):

c_\text{skip}(\sigma) = \frac{\sigma_\text{data}^2}{(\sigma - \sigma_\min)^2 + \sigma_\text{data}^2},\quad c_\text{out}(\sigma) = \frac{\sigma_\text{data}\,(\sigma - \sigma_\min)}{\sqrt{\sigma_\text{data}^2 + \sigma^2}}

Consistency Loss (core): take adjacent noise levels $t_n < t_{n+1}$, and require the student's outputs at $(x_{t_n}, t_n)$ and $(x_{t_{n+1}}, t_{n+1})$ to agree —

$\boxed{\;\mathcal{L}_\text{CD}(\theta) = \mathbb{E}\left[\lambda(t_n)\, d\!\Big(f_\theta(x_{t_{n+1}}, t_{n+1}),\; f_{\theta^-}(\hat x_{t_n}, t_n)\Big)\right]\;}$

• $\theta^-$: EMA target (like BYOL, prevents representation collapse)
• $\hat x_{t_n} = x_{t_{n+1}} - (t_{n+1} - t_n) \cdot v_\phi(x_{t_{n+1}}, t_{n+1})$: teacher one-step ODE reverse
• $d$: L2 or LPIPS (CM paper uses LPIPS for ImageNet 64)
• $\lambda(t_n)$: weight, CM paper takes 1

💡 Key difference CD vs CT — CD (Consistency Distillation) uses a pretrained teacher $v_\phi$ to compute $\hat x_{t_n}$; CT (Consistency Training) has no teacher and uses $\hat x_{t_n} = x_0 + t_n \epsilon$ (the same noise sample with different noise levels). CD with LPIPS + EMA reaches FID 3.55 on CIFAR-10; CT only reaches 8.7 — until iCT caught up.

2.2　Derive the consistency loss from PF-ODE (mandatory derivation)

Consider PF-ODE $\frac{dx}{dt} = v_\phi(x_t, t)$ (teacher). The consistency definition requires self-consistency along trajectories $f_\theta(x_{t+\Delta t}, t+\Delta t) = f_\theta(x_t, t)$. First-order Taylor expansion (along the trajectory):

$f_\theta(x_{t+\Delta t}, t+\Delta t) \approx f_\theta(x_t, t) + \Delta t \cdot \frac{d f_\theta}{dt}$

where $\frac{d f_\theta}{dt} = \partial_t f_\theta + \partial_x f_\theta \cdot v_\phi$. So the continuous-time consistency loss:

$\mathcal{L}_\text{cont} = \mathbb{E}\left\|\frac{d f_\theta}{dt}\right\|^2 = \mathbb{E}\left\|\partial_t f_\theta + (\partial_x f_\theta)\, v_\phi(x_t, t)\right\|^2$

Discretization: use $f_{\theta^-}$ (EMA) as a stop-gradient anchor, $\hat x_{t_n}$ from one teacher ODE step:

$\mathcal{L}_\text{CD} \approx \mathbb{E}\|f_\theta(x_{t_{n+1}}, t_{n+1}) - f_{\theta^-}(\hat x_{t_n}, t_n)\|^2$

⚠️ Can't drop the EMA anchor — if both sides use $\theta$, the loss degenerates to $\|f_\theta - f_\theta\| = 0$ and the network gets no signal. EMA provides a "past self" as supervision, similar to BYOL's collapse-prevention mechanism. The iCT paper (§2.3) proves you can remove EMA under suitable noise schedule + pseudo-Huber loss — this is one of iCT's core contributions.

2.3　iCT / Improved Techniques (Song-Dhariwal 2023, arXiv:2310.14189)

CT (Consistency Training) originally had quality far below CD. iCT improves four things:

Change	original CT	iCT
Target	EMA $\theta^- = \tau \theta^- + (1-\tau) \theta$	direct stop-grad (no EMA)
Loss	LPIPS	Pseudo-Huber $d(a, b) = \sqrt{\lVert a-b \rVert^2 + c^2} - c$
Noise sched	uniform discrete $\sigma_n$	Lognormal: $\log \sigma \sim \mathcal{N}(P_\text{mean}, P_\text{std}^2)$
Step count	fixed $N$	Curriculum: N(k) = \lceil N_\min \cdot (N_\max/N_\min)^{k/K} \rceil

Motivation for pseudo-Huber design:

• LPIPS introduces a bias toward ImageNet pretrained features — at eval FID looks good but there's actual distribution shift
• L2 is sensitive to outliers and unstable to train
• Pseudo-Huber $\sqrt{\|a-b\|^2 + c^2} - c$: for small residuals ≈ $\|a-b\|^2/(2c)$ (L2), for large residuals ≈ $\|a-b\|$ (L1) — adaptively robust

Results: iCT achieves 1-step FID 2.83 / 2-step FID 2.46 on CIFAR-10 (paper abstract numbers; the deeper iCT-deep reaches 2.51 / 2.24), and doesn't depend on a teacher — fully opening the ceiling for from-scratch consistency training.

2.4　sCM / TrigFlow (Lu-Song 2024 OpenAI, arXiv:2410.11081)

Problem: discrete-time CM has two big ailments — (i) discretization error (larger $N$ is more accurate but slower) and (ii) various hyperparameters (noise schedule / EMA decay / loss curriculum) that are finicky to tune.

TrigFlow parametrization: write the forward path in trig form —

$\boxed{\;x_t = \cos(t)\, x_0 + \sin(t)\, z,\quad t \in [0, \pi/2],\; z \sim \mathcal{N}(0, I)\;}$

Boundaries: at $t = 0$, $x_t = x_0$ (data); at $t = \pi/2$, $x_t = z$ (standard Gaussian).

Why trig form? It is the unique parametrization that simultaneously simplifies the following four things (Lu-Song 2024 Theorem 1):

• EDM precond: $D_\theta(x_t, t) = \cos(t)\, x_t - \sin(t)\, F_\theta$, automatically satisfies the boundary
• PF-ODE: $\frac{dx_t}{dt} = -\sin(t) x_0 + \cos(t) z$, clean expression
• CM output: $f_\theta(x_t, t) = \cos(t) x_t - \sin(t) (\sigma_d F_\theta(x_t / \sigma_d, c_\text{noise}(t)))$ ($\sigma_d$ is data std)
• Continuous-time consistency loss: gradient can be written closed-form directly

Continuous-Time Consistency Loss (sCM core): sCM rewrites the continuous-time CM gradient as a stop-gradient MSE surrogate (rather than simplifying the target to $r\cdot\mathrm{JVP}$ — that would become a zero-signal self-reference during warmup at $r=0$). The correct form:

$\mathcal{L}_\text{sCM}(\theta, \phi) = \mathbb{E}_{x, t}\!\left[\frac{e^{w_\phi(t)}}{D}\Big\|F_\theta(x_t/\sigma_d, t) - \operatorname{sg}\!\big(F_{\theta^-}(x_t/\sigma_d, t) + g_{\theta^-}(x_t, t)\big)\Big\|_2^2 - w_\phi(t)\right]$

where $F_{\theta^-}$ is the EMA / stop-grad copy. TrigFlow consistency function: $f_\theta(x_t, t) = \cos t\, x_t - \sin t\, \sigma_d F_\theta(x_t/\sigma_d, t)$. Let $\hat v_t = dx_t/dt$ (in sCT $= \cos t\, z - \sin t\, x_0$; in sCD given by the teacher PF-ODE); the JVP-rearranged tangent target:

$g = -\cos^2(t)\,(\sigma_d F_{\theta^-} - \hat v_t) - r\cos(t)\sin(t)\!\left(x_t + \sigma_d \frac{dF_{\theta^-}}{dt}\right),\quad g \leftarrow \frac{g}{\|g\|_2 + c}.$

The warmup $r: 0 \to 1$ only opens the second term; at $r=0$ the term $-\cos^2(t)(\sigma_d F_{\theta^-} - \hat v_t)$ remains, so it degenerates to velocity / diffusion matching — not zero loss.

Key tricks:

• Adaptive double normalization: normalize both input and output by $\sigma_d$ + $\sigma(t)$ so the network's effective scale doesn't depend on $t$
• Tangent warmup (not "turn off all tangents"): $r$ controls the second term $-r\cos t\sin t(\cdots)$, the first term is always active; adaptive weighting $w_\phi(t)$ together with tangent normalization reduces variance
• JVP via forward-mode autodiff: PyTorch torch.func.jvp, about 2× faster than computing the Jacobian via backward

Results: 1.5B parameters, ImageNet 512×512 2-step FID 1.88, within 10% of the best diffusion baseline — first time CM hits top-tier numbers at large-scale high-res.

2.5　CTM / Consistency Trajectory Models (Kim et al. 2024 ICLR, arXiv:2310.02279)

Problem: CM can only map (x_t, t) \to x_{\sigma_\min} (trajectory endpoint); cannot jump to intermediate points; step count is fixed.

CTM's extension: learn $G(x_t, t, s)$ — jump from $(x_t, t)$ to any $s < t$:

$G_\theta(x_t, t, s) \approx \text{ODE-solver}(x_t, t \to s)$

• s = \sigma_\min degenerates to CM
• $s = t$ degenerates to identity
• Intermediate $s$ lets the user freely choose NFE: 3-step = $G(z, T, t_1) \to G(\cdot, t_1, t_2) \to G(\cdot, t_2, 0)$

Loss: trajectory matching —

$\mathcal{L}_\text{CTM} = \mathbb{E}\Big[d\big(G_\theta(x_t, t, s),\; \text{ODE-solver}^\text{teacher}(x_t, t \to s)\big)\Big] + \lambda\, \mathcal{L}_\text{score}$

• First term: trajectory consistency, the student reproduces teacher ODE
• Second term: auxiliary score matching (prevents trivial solutions)

Results: CIFAR-10 1-step FID 1.73, ImageNet 64 1.92 — SOTA. Core contribution: turns step count from "hard-coded" to "runtime selectable".

2.6　LCM / Latent Consistency Models (Luo et al. 2023, arXiv:2310.04378)

LCM = CM on latent diffusion (SD 1.5 / SDXL). Three improvements:

1. Latent space: do CM in VAE latent ($f=8$), saving $64\times$ compute
2. CFG distilled into the student: randomly sample guidance scale w \in [w_\min, w_\max] during training and feed $w$ as an extra condition — $f_\theta(x_t, t, c, w)$. No double forward needed at inference for conditional + unconditional
3. Skipping-Step Distillation: use a $k$-step skipped teacher (e.g., $k=20$ skipping to 50/20 ≈ 2.5) to accelerate convergence

Results: 4-step SD-XL generation, FID close to 50-step SDXL (same base model).

2.7　LCM-LoRA (Luo et al. 2023, arXiv:2311.05556)

Core idea: the LCM-trained "weight difference" $\Delta \theta = \theta_\text{LCM} - \theta_\text{SD}$ can be parametrized as a LoRA —

$\Delta W = B A,\quad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \in \{8, 16, 32, 64\}$

Only train $A, B$ (~22M params / SDXL); merge as $W' = W + \alpha B A$.

✅ Ecosystem value of LCM-LoRA — the SD 1.5 / SDXL ecosystem has tens of thousands of fine-tuned models (DreamShaper / RealisticVision / various character LoRAs). LCM-LoRA doesn't require retraining each base model; users just attach LCM-LoRA + their own LoRA and generate in 4 steps. This is why LCM is far more ubiquitous in production than DMD/ADD.

2.8　TCD / Trajectory Consistency Distillation (Zheng et al. 2024, arXiv:2402.19159)

TCD = LCM + trajectory-aware improvements. Two main contributions:

1. Trajectory consistency function: relax the boundary condition from "single \sigma_\min" to "any point along the trajectory". Specifically use a semi-linear consistency function (derived via exponential integrator) to reduce parametrization error
2. Strategic stochastic sampling: during multi-step inference, explicitly control stochasticity — add controllable perturbation via $\gamma \in [0, 1]$ to avoid accumulated error skewing the distribution

Practical effect: low-NFE (4 step) quality is higher than LCM; at high NFE (8+ step) it's even more detailed than the teacher itself (the stochastic sampling adds expressivity).

2.9　rCM / Score-Regularized Continuous-Time CM (2025, arXiv:2510.08431)

📍 One of the latest CM works at the time of writing (2025-2026) — rCM = "Score-Regularized Continuous-Time Consistency Model", arXiv:2510.08431 verified.

Motivation: sCM has a quality bottleneck on fine details — the authors attribute it to the mode-covering nature of forward divergence (KL(p_data ‖ p_student) tends to cover all modes, blurring details).

rCM approach: add a score distillation regularizer (reverse-divergence flavor, similar to DMD's KL gradient) on top of the sCM loss, so the student has both mode-seeking (sharp details) + mode-covering (diversity).

Results: on Cosmos-Predict2, Wan 2.1 (14B), produces 5-second video in 1-4 steps with quality matching DMD2 but better diversity.

§3 Distribution Matching Distillation (DMD family)

3.1　DMD core: reverse-KL via score gap (Yin et al. 2024 CVPR, arXiv:2311.18828)

Problem view: student $G_\theta$ maps noise directly to image; we want its output distribution $p_\text{fake}$ to match the teacher distribution $p_\text{real}$. Directly optimize the gradient of $\text{KL}(p_\text{fake} \| p_\text{real})$ —

$\nabla_\theta \text{KL}(p_\text{fake}^\theta \| p_\text{real}) = -\mathbb{E}_{x \sim p_\text{fake}^\theta}\!\left[\big(\nabla_x \log p_\text{real}(x) - \nabla_x \log p_\text{fake}(x)\big) \cdot \frac{\partial G_\theta}{\partial \theta}\right]$

Key observation: $\nabla_x \log p_\text{real}$ is the teacher score $s_\text{real}$ (teacher diffusion model is off-the-shelf); $\nabla_x \log p_\text{fake}$ uses a fake score model $s_\text{fake}$ (a small diffusion trained on the student's current outputs).

DMD Loss (two losses trained jointly; strict notation: $\mu$ is the denoiser / mean predictor, $s_\mu(x_t,t) = (\alpha_t \mu(x_t,t) - x_t)/\sigma_t^2$ is the score derived from the denoiser; the DMD paper uses the denoiser, not the raw score):

$$\boxed{\; \begin{aligned} \nabla_\theta \mathcal{L}_\text{DMD}^G &= \mathbb{E}_{z, t, \epsilon}\!\left[w_t\,\alpha_t\,(s_\text{fake}(x_t, t) - s_\text{real}(x_t, t))^\top\,\tfrac{\partial G_\theta(z)}{\partial\theta}\right] \quad\text{// student, surrogate} \\ \mathcal{L}_\text{fake}(\phi_f) &= \mathbb{E}\!\left[\lambda_t\,\|\mu_{\phi_f}(x_t, t) - \operatorname{sg}(G_\theta(z))\|_2^2\right] \quad\text{// fake denoiser DSM target is the student's current output} \end{aligned} \;}$$

where $x_t = \alpha_t G_\theta(z) + \sigma_t \epsilon$. Auxiliary regression loss: DMD v1 also adds $\mathbb{E}\|G_\theta(z) - \text{ODE-solver}^\text{teacher}(z)\|^2$ (teacher-pair supervision) to keep the student from drifting — but this requires pre-generating a large pool of teacher pairs, expensive and mode-limited; DMD2 drops this term.

3.2　Deriving the DMD gradient from reverse-KL (mandatory derivation)

Let student $G_\theta(z) \mapsto x$ with $z \sim \mathcal{N}(0, I)$. The fake distribution is $p_\text{fake}^\theta(x) = G_\theta \# \mathcal{N}(0, I)$ (push-forward).

Reverse KL:

$\text{KL}(p_\text{fake} \| p_\text{real}) = \mathbb{E}_{x \sim p_\text{fake}}[\log p_\text{fake}(x) - \log p_\text{real}(x)]$

Take $\nabla_\theta$:

$\nabla_\theta \text{KL} = \mathbb{E}_z\!\left[\nabla_\theta \log p_\text{fake}^\theta(G_\theta(z)) - \nabla_\theta \log p_\text{real}(G_\theta(z))\right]$

Chain rule on the second term: $\nabla_\theta \log p_\text{real}(G_\theta(z)) = \nabla_x \log p_\text{real}(x)\big|_{x=G_\theta(z)} \cdot \partial G_\theta / \partial \theta$.

Expand the first term and use $\mathbb{E}_{p_\text{fake}}[\nabla_\theta \log p_\text{fake}] = 0$ (score-function trick); rearrange:

$\nabla_\theta \text{KL} = -\mathbb{E}_z\!\left[(\nabla_x \log p_\text{real} - \nabla_x \log p_\text{fake})\big|_{x=G_\theta(z)} \cdot \partial_\theta G_\theta(z)\right]$

But $p_\text{real}, p_\text{fake}$ in high dimensions have scores that are discontinuous / non-smooth — DMD's trick is to compute scores at all noise levels $t$ for $x_t = G_\theta(z) + \sigma_t \epsilon$, moving estimation to the smooth $p_t$ — this is why DMD needs both a real diffusion teacher and a fake diffusion (both provide "score at different noise levels").

💡 DMD vs CM fundamental difference — CM is trajectory matching (make the student reproduce the teacher ODE solution); DMD is distribution matching (make the two distributions have equal scores everywhere). CM requires step alignment (noise schedules align), DMD does not — DMD's student can be any differentiable generator architecture.

3.3　DMD2 (Yin et al. 2024 NeurIPS, arXiv:2405.14867)

Four improvements:

Change	DMD	DMD2
Regression loss	requires teacher pairs (expensive)	dropped
GAN	❌	add GAN classifier: attached to the fake-diffusion denoiser bottleneck, discriminating on noised real / noised fake (not clean images)
TTUR	1:1	fake denoiser updated ~5× per generator step (paper default 5:1 on ImageNet)
Student	1-step only	multi-step backward simulation: during training, run the current student per the inference schedule to obtain intermediate noisy states, then compute DMD/GAN losses on those states to align training/inference distributions

DMD2 total loss (generator side):

$\mathcal{L}_\text{DMD2}^G = \underbrace{\mathcal{L}_\text{DMD}^G(\theta)}_{\text{score gap surrogate}} + \lambda_\text{GAN} \cdot \mathcal{L}_\text{adv}^G(\theta)$

Discriminator side: DMD2's D is typically a classifier head on top of the fake-denoiser bottleneck, with input being noised image $x_t = \alpha_t x + \sigma_t\epsilon$ (not clean $x$):

$\mathcal{L}_D = \mathbb{E}_{x \sim p_\text{data}, t}\!\left[\text{softplus}(-D_\psi(x_t, t))\right] + \mathbb{E}_{z, t}\!\left[\text{softplus}(D_\psi(\hat x_t^{\text{fake}}, t))\right]$

Multi-step backward simulation (key, much stronger than simple unrolling): during training run the current student per a $K$-step inference schedule to obtain intermediate noisy states $x_{t_k}$, then call the student / compute DMD-GAN loss on these states. This ensures the training input distribution = the input distribution seen by the student at step $k$ at inference, not just naively noised real images.

Results: ImageNet 64 1-step FID 1.28 (DMD v1 was 2.62), the first time one-step diffusion surpassed GANs. Production: DMD2-SDXL produces 1-step 1024×1024 megapixel images.

3.4　Statistical-physics intuition for the score gap

The reverse-KL "score gap" $s_\text{real} - s_\text{fake}$ physically corresponds to the force difference between two Gibbs distributions —

$s_\text{real} - s_\text{fake} = \nabla_x \log\frac{p_\text{real}}{p_\text{fake}} = -\nabla_x [V_\text{real}(x) - V_\text{fake}(x)]$

Treating the student as a particle, $s_\text{real} - s_\text{fake}$ is the "force" pushing it from $p_\text{fake}$ toward $p_\text{real}$. This is the fundamental difference between DMD and GAN — GANs use a discriminator's binary signal; DMD uses the score gap to give a dense vector-field signal, with far higher sample efficiency.

§4 Adversarial Distillation (ADD / LADD family)

4.1　ADD / SDXL-Turbo (Sauer et al. 2023, arXiv:2311.17042)

Stability AI 2023.11, making SDXL produce 512² images in 1 step. Two supervisions:

$\boxed{\;\mathcal{L}_\text{ADD} = \mathcal{L}_\text{adv}^G(\theta, \psi) + \lambda \cdot \mathcal{L}_\text{distill}(\theta, \phi)\;}$

• $\mathcal{L}_\text{adv}$: hinge loss + DINOv2 vision backbone as discriminator (not training D from scratch; fix DINOv2 + multiple heads)
• $\mathcal{L}_\text{distill}$: discrete form of "score distillation" — MSE between student 1-step output and teacher multi-step output

The DINOv2 discriminator is key to ADD:

• Ordinary GANs train D from scratch, which is unstable for 1-step generators (severe mode collapse)
• DINOv2 provides "pretrained high-level perceptual features", anchoring the discrimination problem in a strong semantic space
• Multiple heads (different layer features) + hinge loss → stable training

⚠️ Actual form of the distillation loss — the $\mathcal{L}_\text{distill}$ in the ADD paper is a score-distillation style loss (estimating targets using the teacher denoiser on the noised student output), not a plain pixel-space MSE and not a KL. The pedagogical version $\|G_\theta - \text{ODE}\|^2$ shown earlier is an illustrative simplification; see the original paper Eq. (6-7) for details. ADD relies on the GAN loss for mode diversity.

Results: SDXL-Turbo at 512×512 does 1-step in ~100ms/image (A100), CLIP score on par with 4-step SDXL; at 1024×1024 quality is limited on SDXL-Turbo and addressed by later LADD / SD3-Turbo / Lightning.

4.2　LADD / SD3-Turbo (Sauer et al. 2024, arXiv:2403.12015)

Problem: ADD computes distill loss in pixel space and uses DINOv2 D — unfriendly to high resolution (1024+) and latent diffusion; pixel decoding is expensive and DINOv2's input resolution is limited to 224 / 518.

LADD = Latent ADD: move the discriminator directly into latent space —

	ADD	LADD
Discriminator backbone	DINOv2 (pixel)	teacher diffusion's own intermediate layer features (latent)
Distillation	pixel MSE	latent-space distill
Resolution scale	limited by DINOv2	latent any size
Application models	SDXL	SD3 (8B), FLUX (12B)

Discriminator design: extract the teacher's MM-DiT block and fine-tune it as the D backbone — the rationale being intermediate features learned during diffusion training already implicitly encode "what a realistic latent looks like".

Results:

• SD3-Turbo = SD3 8B + LADD → 4-step 1024² rivals multi-step SD3
• FLUX.1-schnell = FLUX 12B + LADD → 1-4 step 1024² generation (Apache 2.0 open source)

4.3　SDXL-Lightning (Lin et al. 2024, arXiv:2402.13929)

ByteDance's open-source SDXL distillation, with progressive + adversarial two-pronged design:

• Progressive (halving): starting from teacher multi-step, halve the step count per stage ($T \to T/2 \to T/4 \to \dots$), each stage fitting the previous stage's teacher with MSE (in the Salimans-Ho 2022 progressive distillation lineage). Finally selectable at 1/2/4/8-step
• Adversarial: end each stage with GAN loss for fidelity
• Discriminator: self-trained (not borrowed from an off-the-shelf backbone like ADD/LADD)

Results: SDXL 1024² selectable at 1-step / 2-step / 4-step, open-source LoRA form (similar to LCM-LoRA), ecosystem-friendly.

4.4　ADD/LADD/Lightning comparison

Method	Discriminator	Distill loss	Application	1-step quality
ADD	DINOv2 (pixel)	pixel MSE	SDXL	medium (512²)
LADD	teacher MM-DiT feat (latent)	latent score-distill	SD3, FLUX	high (1024²)
Lightning	self-trained CNN	progressive MSE	SDXL	medium-high
DMD2	self-trained + score gap	reverse-KL via score	SDXL	high (with diversity)

💡 Production selection cheat sheet — if base is SD 1.5 / SDXL use LCM-LoRA (largest ecosystem) or SDXL-Lightning (open-source stable); if base is SD3 / FLUX, LADD is the official route; for GAN-free + score-based pick DMD2; for academic SOTA chasing, pick sCM / rCM.

§5 Flow / Rectified Flow distillation

5.1　Rectified Flow + Reflow route (Liu et al. 2022, arXiv:2209.03003)

Rectified Flow path: $x_t = (1-t) x_0 + t\, x_1$, $x_0 \sim \mathcal{N}(0, I)$ (noise side), $x_1 \sim p_\text{data}$, target $u_t = x_1 - x_0$ (constant vector).

Reflow algorithm (reverse via ODE, re-pair, re-train):

1. Train $v_\theta^{(1)}$ on independent pairs $(x_0, x_1) \sim p_0 \otimes p_\text{data}$
2. Use $v_\theta^{(1)}$ to run the ODE and generate coupled pairs $(x_0, x_1^{(1)})$, i.e., $x_1^{(1)} = x_0 + \int_0^1 v_\theta^{(1)}(x_t, t)\, dt$
3. Retrain $v_\theta^{(2)}$ on coupled pairs — the new trajectory is straighter (transport-cost-non-increasing theorem)

Why does reflow make trajectories straight?

Consider the transport cost $\mathbb{E}[\|x_1 - x_0\|^2]$ as coupling. Independent pairs have large cost; after reflow $(x_0, x_1^{(1)})$ has already been naturally paired by the ODE — it is the "optimal transport" under the current $v_\theta^{(1)}$. Liu 2022 shows: after another round of training the total transport cost is non-increasing (in fact often strictly decreasing), and "the curve being straight" is equivalent to the vector field not depending on $t$ — $v(t, x) =$ const along the trajectory → 1-step generation.

InstaFlow (2023, arXiv:2309.06380): the first to apply reflow to SD, 1-step generation FID 23.3 (512²).

5.2　The "straight-line" limit of reflow

Ideal limit: if reflow converges to fully straight, then $v_\theta(t, x) = v_\theta(x)$ (independent of $t$), and 1-step Euler suffices —

$x_1 = x_0 + 1 \cdot v_\theta(x_0)$

In practice: 1-2 rounds of reflow already make trajectories straight enough to support 4-step Euler matching 50-step quality; full 1-step requires more reflow + adversarial fine-tuning (like SD3-Turbo / FLUX-schnell).

5.3　SD3-Turbo / FLUX-schnell = RF + LADD

The actual production stack:

SD3 / FLUX (Rectified Flow, ~50-step 1024²)
  │ pretrain
  ↓
LADD distillation (teacher distill + latent discriminator)
  ↓
SD3-Turbo / FLUX-schnell (1-4 step 1024²)

Not pure reflow — LADD provides adversarial fidelity, more stable than pure reflow for high-res production.

5.4　Flow-OPD (arXiv:2605.08063, 2026) — out-of-scope sidebar

📍 Scope clarification — Flow-OPD's main contribution is multi-reward RL alignment + on-policy specialist distillation, not few-step inference distillation. It appears here because its name contains "Distillation" and it involves flow models; but the core thread of §2-§4 (CM/DMD/ADD compressing 50 steps to 1-4) differs from Flow-OPD's RL-alignment objective.

The companion piece diffusion_post_training_tutorial.md has a full discussion of RL alignment (Flow-GRPO / Diffusion-DPO / DDPO etc.); Flow-OPD is more accurately placed there.

Brief idea (sidebar only, see post-training tutorial + original paper for depth): use multiple reward-specific teachers (each GRPO fine-tuned on a reward) for on-policy distillation supervision; the student remains few-step at inference while simultaneously gaining multi-reward alignment.

Loss form (simplified sketch, refer to original paper):

$\mathcal{L}_\text{OPD-sketch} = \mathbb{E}_{x \sim \pi_\theta}\!\left[\sum_k w_k(x) \cdot \|v_\theta(x_t, t) - v_{\phi_k}(x_t, t)\|^2\right]$

with task-aware weighting $w_k$; this is only a structural sketch — do not claim that Flow-OPD and DMD are "degeneratively equivalent" mathematically (no reliable basis for that claim; please don't assert it in interviews).

Paper-reported results: based on SD 3.5 Medium, GenEval 63 → 92, OCR 59 → 94 (see paper Table for details).

5.5　Rectified Diffusion / follow-up work

📍 Brief mention of related work outside main scope, Rectified Diffusion (arXiv:2410.07303)

Follow-up work (e.g., Rectified Diffusion) challenges whether straightness is necessary — finding that a straight line is not a necessary condition; it suffices for the ODE solution space to be sufficiently expressive. This line and sCM's "continuous-time CM" show signs of mathematical convergence.

§6 CFG distillation

6.1　Why CFG must be distilled separately

CFG inference:

$\tilde\epsilon(x, c) = (1 + w) \epsilon_\theta(x, c) - w\, \epsilon_\theta(x, \emptyset)$

Two forwards per step (conditional + unconditional), doubling latency. So production CFG-aware models need to distill CFG into a single forward.

6.2　Guidance Distillation (Meng et al. 2023 CVPR, arXiv:2210.03142)

Stage 1 — Guidance distillation: train a student $\tilde\epsilon_\theta(x, c, w)$ that takes guidance scale $w$ as an extra condition, directly outputting the post-CFG score:

$\mathcal{L}_\text{guide} = \mathbb{E}\!\left[\|\tilde\epsilon_\theta(x_t, c, w) - \tilde\epsilon^*(x_t, c, w)\|^2\right]$

where $\tilde\epsilon^*$ is the CFG output obtained by the teacher's two explicit forwards. The student does only one forward.

Stage 2 — Step distillation: on top of stage 1, stack progressive distillation, compressing steps from 32 to 4 → 2 → 1.

LCM's CFG-aware design is inherited from this — feeding $w$ as a condition is the key to LCM-LoRA.

6.3　Step-distillation vs trajectory-distillation differences

	Step-distillation (Salimans-Ho 2022)	Trajectory-distillation (CM/CTM)
Goal	distill $N$-step student to $N/2$-step	learn trajectory function $f(x_t, t) \to x_0$
Training stage	multi-stage progressive	single stage
Step count	halve each round (32→16→8→4→2→1)	arbitrary (1-step direct training)
Boundary condition	none required	requires f(x_{\sigma_\min}, \sigma_\min) = x
Teacher	previous-stage student (self-distillation)	original diffusion

💡 Historical arc — 2022 progressive distillation was the first to make 4-step diffusion usable; 2023 CM directly enabled 1-step via trajectory function; 2024 sCM / DMD2 pushed 1-step to SOTA. Idea evolution: iterative approximation (progressive) → function fitting (CM) → distribution matching (DMD) → trig parametrization (sCM).

§7 From-scratch PyTorch code

7.1　Code 1: Consistency Distillation Loss (CD, base CM)

import torch
import torch.nn as nn
import torch.nn.functional as F

def edm_precond(F_net, x, sigma, sigma_data=0.5, sigma_min=0.002):
    """EDM-style precond, so boundary f(x, sigma_min) = x is automatically satisfied"""
    c_skip = sigma_data**2 / ((sigma - sigma_min)**2 + sigma_data**2)
    c_out  = sigma_data * (sigma - sigma_min) / torch.sqrt(sigma_data**2 + sigma**2)
    c_in   = 1.0 / torch.sqrt(sigma_data**2 + sigma**2)
    c_noise = 0.25 * torch.log(sigma)
    # Broadcast to [B, 1, 1, 1] (images)
    c_skip = c_skip.view(-1, 1, 1, 1)
    c_out  = c_out.view(-1, 1, 1, 1)
    c_in   = c_in.view(-1, 1, 1, 1)
    F_x = F_net(c_in * x, c_noise)
    return c_skip * x + c_out * F_x

@torch.no_grad()
def teacher_ode_step(x_t1, t1, t0, teacher):
    """teacher one-step Heun (EDM 2nd-order) reverse: t1 -> t0"""
    d1 = (x_t1 - teacher(x_t1, t1)) / t1  # current gradient
    x_euler = x_t1 + (t0 - t1) * d1       # Euler prediction
    d2 = (x_euler - teacher(x_euler, t0)) / t0
    return x_t1 + 0.5 * (t0 - t1) * (d1 + d2)

def consistency_distillation_loss(student, student_ema, teacher,
                                   x_0, sigmas, N=18):
    """
    student / student_ema: same architecture; ema is a stop-grad copy
    teacher: pretrained diffusion (EDM denoiser)
    x_0: clean image batch [B, C, H, W]
    sigmas: noise schedule, **indices in ascending = noise increasing** (sigmas[0]=sigma_min, sigmas[N]=sigma_max)
    
    !!! Pedagogical sketch: we convention sigmas in ascending order so t_{n+1} > t_n is convenient.
        For production refer to EDM official code (karras/edm: typically descending sigmas) + paper Eq. form.
    """
    B = x_0.shape[0]
    # 1) Random adjacent noise level n ~ U{0, N-1}
    n = torch.randint(0, N, (B,), device=x_0.device)
    t_n1 = sigmas[n + 1]    # higher noise (per the above convention: sigmas ascending)
    t_n  = sigmas[n]        # lower noise
    
    # 2) Sample x_{t_{n+1}} = x_0 + t_{n+1} * eps
    eps = torch.randn_like(x_0)
    x_tn1 = x_0 + t_n1.view(-1, 1, 1, 1) * eps
    
    # 3) Teacher one-step ODE reverse to get x_{t_n}
    with torch.no_grad():
        x_tn = teacher_ode_step(x_tn1, t_n1, t_n, teacher)
    
    # 4) student / student_ema both pass through EDM precond
    f_online = edm_precond(student, x_tn1, t_n1)
    with torch.no_grad():
        f_target = edm_precond(student_ema, x_tn, t_n)
    
    # 5) consistency loss (LPIPS / L2; use L2 here)
    loss = F.mse_loss(f_online, f_target)
    return loss

def update_ema(ema_model, model, decay=0.9999):
    """EMA target, BYOL-style"""
    with torch.no_grad():
        for p_ema, p in zip(ema_model.parameters(), model.parameters()):
            p_ema.data.mul_(decay).add_(p.data, alpha=1 - decay)

7.2　Code 2: iCT (no EMA + Pseudo-Huber + Lognormal + Curriculum)

def pseudo_huber(a, b, c=0.00054):
    """Pseudo-Huber loss: sqrt(||a-b||^2 + c^2) - c
    Small residual ≈ L2/2c, large residual ≈ L1. iCT paper c=0.00054 (CIFAR-10)"""
    return torch.sqrt((a - b).pow(2).sum(dim=(1, 2, 3)) + c**2).mean() - c

def lognormal_sigma(B, P_mean=-1.1, P_std=2.0, sigma_min=0.002, sigma_max=80.0):
    """iCT samples sigma from lognormal instead of uniform
    log_sigma ~ N(P_mean, P_std)"""
    log_sigma = torch.randn(B) * P_std + P_mean
    sigma = torch.exp(log_sigma).clamp(sigma_min, sigma_max)
    return sigma

def get_curriculum_N(step, total_steps, N_min=10, N_max=1280, schedule='exp'):
    """Step-count curriculum: N grows from 10 to 1280
    Over K training steps, N(k) = ceil(N_min * (N_max/N_min)^(k/K))"""
    k = step / total_steps
    if schedule == 'exp':
        N = N_min * (N_max / N_min) ** k
    else:
        N = N_min + (N_max - N_min) * k
    return int(math.ceil(N))

def ict_loss(student, x_0, step, total_steps):
    """iCT: no EMA, no LPIPS, no teacher
    consistency loss on (x_0 + sigma_n*eps, x_0 + sigma_{n+1}*eps) with SAME eps"""
    B = x_0.shape[0]
    device = x_0.device
    
    # 1) curriculum N
    N = get_curriculum_N(step, total_steps)
    
    # 2) Pick adjacent sigma_n, sigma_{n+1} (chosen from N+1 lognormal-discretized points)
    # !!! convention: sigmas ascending, so sigmas[n+1] > sigmas[n] (consistent with §2.3 CD code)
    sigmas = lognormal_sigma(N + 1).to(device).sort(descending=False).values
    n_idx = torch.randint(0, N, (B,), device=device)
    t_n1 = sigmas[n_idx + 1]  # higher noise
    t_n  = sigmas[n_idx]      # lower noise
    
    # 3) Key: same epsilon, two different noise levels (no teacher needed)
    eps = torch.randn_like(x_0)
    x_tn1 = x_0 + t_n1.view(-1, 1, 1, 1) * eps
    x_tn  = x_0 + t_n.view(-1, 1, 1, 1) * eps
    
    # 4) Student runs both (no EMA; target uses stop_grad)
    f_online = edm_precond(student, x_tn1, t_n1)
    with torch.no_grad():
        f_target = edm_precond(student, x_tn, t_n)
    
    # 5) Pseudo-Huber
    loss = pseudo_huber(f_online, f_target, c=0.00054)
    return loss

7.3　Code 3: sCM Continuous-Time Loss (TrigFlow)

import torch.func as tfunc

def trigflow_xt(x_0, z, t):
    """TrigFlow path: x_t = cos(t) x_0 + sin(t) z, t in [0, π/2]"""
    cos_t = torch.cos(t).view(-1, 1, 1, 1)
    sin_t = torch.sin(t).view(-1, 1, 1, 1)
    return cos_t * x_0 + sin_t * z

def scm_loss(F_net, x_0, sigma_data=0.5, r_warmup=0.5):
    """
    Simplified continuous-time CM loss (sCM, Lu-Song 2024).
    F_net: student network F_θ(x_t / σ_d, t)
    r_warmup: NCS warmup ratio (0=pure score, 1=pure CM)
    """
    import math
    B = x_0.shape[0]
    device = x_0.device

    # 1) lognormal t (TrigFlow time t in [0, π/2])
    log_t = torch.randn(B, device=device) * 1.0 - 0.4  # σ ≈ 1, mean shift
    t = torch.sigmoid(log_t) * (math.pi / 2 - 0.001) + 0.001  # avoid boundary

    # 2) Sample x_t
    z = torch.randn_like(x_0)
    x_t = trigflow_xt(x_0, z, t)

    # 3) PF-ODE tangent direction (TrigFlow: dx_t/dt = -sin(t) x_0 + cos(t) z)
    cos_t = torch.cos(t).view(-1, 1, 1, 1)
    sin_t = torch.sin(t).view(-1, 1, 1, 1)
    dxdt = -sin_t * x_0 + cos_t * z

    # 4) F_θ output at (x_t/σ_d, t) + JVP (forward-mode autodiff, faster than backward Jacobian)
    def net_fn(xt_norm, t_):
        return F_net(xt_norm, t_)

    x_t_norm = x_t / sigma_data

    # 4b) Student forward (with grad) to get F_out
    F_out = net_fn(x_t_norm, t)

    # 5) sCM target: stop_grad(F_minus + normalized tangent g)
    # First term (velocity matching) still provides signal at r=0; second term is opened by warmup r.
    # JVP tangent direction = dx/dt (PF-ODE direction); JVP output = dF/dt.
    tangent_x = dxdt / sigma_data        # tangent of (x_t/σ_d) along dx/dt direction
    tangent_t = torch.ones_like(t)       # dt/dt = 1

    with torch.no_grad():
        F_minus, dFdt = tfunc.jvp(
            net_fn,
            (x_t_norm, t),
            (tangent_x, tangent_t),
        )
        # First term: velocity / diffusion matching (signal exists even at r=0)
        g = -(cos_t ** 2) * (sigma_data * F_minus - dxdt)
        # Second term: consistency tangent, opened gradually by warmup (cos·sin factor appears once only)
        g = g - r_warmup * (cos_t * sin_t * x_t + sigma_data * cos_t * sin_t * dFdt)
        # Normalize tangent for stability
        g = g / (g.flatten(1).norm(dim=1).view(-1, 1, 1, 1) + 0.1)
        target = F_minus + g

    # 6) surrogate MSE loss (adaptive w_phi(t) omitted in tutorial code)
    loss = F.mse_loss(F_out, target)
    return loss

7.4　Code 4: DMD Loss (Distribution Matching via Score Gap)

class DMDTrainer:
    """
    DMD v1 (Yin 2024 CVPR): three networks
    - G_θ: 1-step student generator (z -> x)
    - s_real: pretrained teacher diffusion (frozen)
    - s_fake: fake diffusion, trained on G_θ outputs
    """
    def __init__(self, G, s_fake, s_real_frozen, sigma_data=0.5):
        self.G = G
        self.s_fake = s_fake  # trainable
        self.s_real = s_real_frozen  # frozen
        self.opt_G = torch.optim.AdamW(G.parameters(), lr=1e-5)
        self.opt_f = torch.optim.AdamW(s_fake.parameters(), lr=1e-5)
        self.sigma_data = sigma_data

    def student_loss(self, z):
        """DMD student loss: alpha_t * (s_fake - s_real)^T · ∂G/∂θ
        where s_*(x_t,t) = (alpha_t * mu_*(x_t,t) - x_t) / sigma_t^2 is derived from the denoiser.
        EDM/VE convention has alpha_t = 1; VP/DDPM uses scheduler's alpha_t.
        """
        x = self.G(z)               # G_θ(z), 1-step output
        B = x.shape[0]
        # Random noise level
        sigma = torch.exp(torch.randn(B, device=x.device) * 1.6 - 1.0).view(-1, 1, 1, 1)
        eps = torch.randn_like(x)
        alpha = 1.0                     # EDM/VE; for VP use scheduler.alpha(t)
        x_t = alpha * x + sigma * eps   # consistent with paper's x_t = alpha*x + sigma*eps

        with torch.no_grad():
            mu_real = self.s_real(x_t, sigma.squeeze())  # frozen denoiser / mean predictor
            mu_fake = self.s_fake(x_t, sigma.squeeze())  # trainable fake denoiser

            # DMD score gap: s_fake - s_real = α(μ_fake - μ_real)/σ².
            # With DMD weight w_t ∝ σ²/α canceling 1/σ², we get w_t(s_fake - s_real) = α(μ_fake - μ_real).
            # Then apply mean-abs normalization for numerical stability (DMD paper Eq. (8)).
            grad_proxy = alpha * (mu_fake - mu_real)
            grad_proxy = grad_proxy / (
                (x.detach() - mu_real).abs().mean(dim=(1, 2, 3), keepdim=True) + 1e-6
            )

        # surrogate: loss = +(x · grad_proxy.detach()).sum(),
        # backward gives ∇L = grad_proxy · ∂G/∂θ = ∇_θ KL(p_fake‖p_real),
        # optimizer step θ -= η∇L thus minimizes KL.
        loss_G = (x * grad_proxy.detach()).sum(dim=(1, 2, 3)).mean()
        return loss_G

    def fake_score_loss(self, z):
        """fake denoiser uses DSM; target is the student's current output (not -eps/sigma score-head form)"""
        with torch.no_grad():
            x = self.G(z)               # detached student output
        B = x.shape[0]
        sigma = torch.exp(torch.randn(B, device=x.device) * 1.6 - 1.0).view(-1, 1, 1, 1)
        eps = torch.randn_like(x)
        alpha = 1.0
        x_t = alpha * x + sigma * eps

        pred_x0 = self.s_fake(x_t, sigma.squeeze())   # denoiser output
        target_x0 = x.detach()                          # student output as DSM target
        loss_f = ((pred_x0 - target_x0) ** 2).flatten(1).mean(1).mean()
        return loss_f

    def step(self, z_batch):
        # 1) update fake score (on G's current output)
        self.opt_f.zero_grad()
        loss_f = self.fake_score_loss(z_batch)
        loss_f.backward()
        self.opt_f.step()
        
        # 2) update G via score gap
        self.opt_G.zero_grad()
        loss_G = self.student_loss(z_batch)
        loss_G.backward()
        self.opt_G.step()
        return loss_G.item(), loss_f.item()

7.5　Code 5: DMD2 Loss (drop regression + GAN + multi-step)

class DMD2Trainer(DMDTrainer):
    """DMD2: drop regression + add noised-input GAN + multi-step backward simulation.
    In the original paper D is a classifier head on the fake-denoiser bottleneck (shared backbone);
    input is the noised image x_t = alpha*x + sigma*eps. Here we use a standalone D as a pedagogical approximation.
    TTUR: fake denoiser + D updated ~5× per generator step (paper default 5:1 on ImageNet).
    """
    def __init__(self, G, s_fake, s_real_frozen, D, sigma_data=0.5,
                 lambda_gan=1.0, num_steps_train=4, ttur_ratio=5):
        super().__init__(G, s_fake, s_real_frozen, sigma_data)
        self.D = D
        self.opt_D = torch.optim.AdamW(D.parameters(), lr=1e-5)
        self.lambda_gan = lambda_gan
        self.K = num_steps_train
        self.ttur_ratio = ttur_ratio

    def _sample_multistep(self, z, K=None, with_grad=False):
        """Backward simulation: run student per inference schedule, **return each step's clean denoised output**.
        with_grad=True keeps grad along the whole chain (for generator loss); False detaches for D / fake denoiser.
        Returns:
          x_finals: list of [B, ...] clean denoised outputs (including final, len = K)
          x_noised_inputs: list of [B, ...] noisy inputs fed to G at the next step (len = K, first = z)
        In the paper this re-noises per EDM/TrigFlow schedule; here sigma_next = t_next as a placeholder.
        """
        K = K or self.K
        ts = torch.linspace(1.0, 0.0, K + 1, device=z.device)
        x_finals = []
        x_noised_inputs = []
        x_input = z
        ctx = torch.enable_grad() if with_grad else torch.no_grad()
        with ctx:
            for k in range(K):
                t_k = ts[k].expand(z.shape[0])
                x_noised_inputs.append(x_input)
                x_clean = self.G(x_input, t_k)   # denoised output
                x_finals.append(x_clean)
                if k < K - 1:
                    sigma_next = ts[k + 1]
                    # re-noise clean output to the next timestep's noisy state
                    x_input = x_clean + sigma_next * torch.randn_like(x_clean)
        return x_finals, x_noised_inputs

    def student_loss_dmd2(self, z, real_batch):
        # 1) backward simulation with grad: get each step's clean denoised output
        x_finals, _ = self._sample_multistep(z, K=self.K, with_grad=True)
        # 2) DMD score gap: compute and average over each clean output (DMD2 paper multi-step loss)
        loss_score = sum(self._score_gap_loss(x_c) for x_c in x_finals) / len(x_finals)
        # 3) GAN generator loss: D discriminates on the noised version of the final clean output
        x_final = x_finals[-1]
        sigma = torch.exp(torch.randn(x_final.shape[0], device=x_final.device) * 1.6 - 1.0).view(-1, 1, 1, 1)
        x_fake_t = x_final + sigma * torch.randn_like(x_final)
        d_fake_logit = self.D(x_fake_t, sigma.squeeze())
        loss_adv = F.softplus(-d_fake_logit).mean()  # non-saturating
        return loss_score + self.lambda_gan * loss_adv

    def fake_score_loss(self, z):
        """Override: fake denoiser DSM target = student's multi-step backward-simulated outputs.
        DMD2 wants the fake denoiser to learn the whole simulated distribution of the generator, not just 1-step G(z).
        """
        with torch.no_grad():
            x_finals, _ = self._sample_multistep(z, K=self.K, with_grad=False)
        # Do DSM on all K clean outputs
        loss_total = 0.0
        for x_c in x_finals:
            B = x_c.shape[0]
            sigma = torch.exp(torch.randn(B, device=x_c.device) * 1.6 - 1.0).view(-1, 1, 1, 1)
            alpha = 1.0
            x_t = alpha * x_c + sigma * torch.randn_like(x_c)
            pred_x0 = self.s_fake(x_t, sigma.squeeze())
            target_x0 = x_c.detach()
            loss_total = loss_total + ((pred_x0 - target_x0) ** 2).flatten(1).mean(1).mean()
        return loss_total / len(x_finals)

    def step(self, z_batch, real_batch):
        """DMD2 step: each generator update pairs with ttur_ratio fake-denoiser + D updates."""
        for _ in range(self.ttur_ratio):
            self.opt_f.zero_grad()
            loss_f = self.fake_score_loss(z_batch)
            loss_f.backward()
            self.opt_f.step()

            self.opt_D.zero_grad()
            loss_D = self.discriminator_loss(z_batch, real_batch)
            loss_D.backward()
            self.opt_D.step()
        # generator update
        self.opt_G.zero_grad()
        loss_G = self.student_loss_dmd2(z_batch, real_batch)
        loss_G.backward()
        self.opt_G.step()
        return loss_G.item(), loss_f.item(), loss_D.item()

    def _score_gap_loss(self, x_fake):
        # Reuse the denoiser-based score gap from DMDTrainer.student_loss (see §7.4);
        # only difference: input is multi-step student output rather than 1-step.
        B = x_fake.shape[0]
        sigma = torch.exp(torch.randn(B, device=x_fake.device) * 1.6 - 1.0).view(-1, 1, 1, 1)
        alpha = 1.0
        x_t = alpha * x_fake + sigma * torch.randn_like(x_fake)
        with torch.no_grad():
            mu_real = self.s_real(x_t, sigma.squeeze())
            mu_fake = self.s_fake(x_t, sigma.squeeze())
            # See §7.4 derivation: w_t(s_fake - s_real) = α(μ_fake - μ_real)
            grad_proxy = alpha * (mu_fake - mu_real)
            grad_proxy = grad_proxy / (
                (x_fake.detach() - mu_real).abs().mean(dim=(1, 2, 3), keepdim=True) + 1e-6
            )
        return (x_fake * grad_proxy.detach()).sum(dim=(1, 2, 3)).mean()

    def discriminator_loss(self, z, real_batch):
        """D discriminates noised image as real vs student output.
        Softplus / non-saturating loss + shared fake-denoiser backbone (we use a separate D for teaching).
        """
        x_finals, _ = self._sample_multistep(z, with_grad=False)
        x_fake = x_finals[-1]
        B = real_batch.shape[0]
        sigma = torch.exp(torch.randn(B, device=real_batch.device) * 1.6 - 1.0).view(-1, 1, 1, 1)
        x_real_t = real_batch + sigma * torch.randn_like(real_batch)
        x_fake_t = x_fake.detach() + sigma * torch.randn_like(x_fake)
        d_real = self.D(x_real_t, sigma.squeeze())
        d_fake = self.D(x_fake_t, sigma.squeeze())
        return F.softplus(-d_real).mean() + F.softplus(d_fake).mean()

7.6　Code 6: ADD (Adversarial Diffusion Distillation, SDXL-Turbo style)

import torchvision  # for DINOv2 backbone

class ADDTrainer:
    """ADD (Sauer 2023): pretrained DINOv2 backbone as discriminator"""
    def __init__(self, G, teacher_diffusion, sigma_data=0.5,
                 lambda_distill=1.0):
        self.G = G
        self.teacher = teacher_diffusion  # frozen
        # DINOv2 backbone + multiple discriminator heads
        self.dino = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14')
        self.dino.eval()
        for p in self.dino.parameters():
            p.requires_grad = False
        # multi-layer head: extract features from different DINOv2 blocks, each with a 1x1 conv head
        self.disc_heads = nn.ModuleList([
            nn.Sequential(nn.Conv1d(1024, 1, 1), nn.Flatten())
            for _ in range(4)
        ])
        self.opt_G = torch.optim.AdamW(G.parameters(), lr=1e-5)
        self.opt_D = torch.optim.AdamW(self.disc_heads.parameters(), lr=1e-5)
        self.lambda_distill = lambda_distill

    def get_dino_features(self, x):
        """Extract features from multiple DINOv2 layers"""
        # Simplified: dinov2_vitl14 intermediate layer hooks (real code uses register_forward_hook)
        # Here we return a list of features for each disc head
        x_resized = F.interpolate(x, size=224, mode='bilinear')
        # mock: assume backbone yields [B, 1024, N_patch] sequence, 4 layers
        feats = self.dino.get_intermediate_layers(x_resized, n=4)
        return feats

    def adv_loss_G(self, x_fake):
        feats = self.get_dino_features(x_fake)
        loss = 0
        for feat, head in zip(feats, self.disc_heads):
            logit = head(feat.transpose(1, 2))  # [B, ?]
            loss += -logit.mean()  # non-saturating
        return loss / len(self.disc_heads)

    def adv_loss_D(self, x_real, x_fake):
        feats_real = self.get_dino_features(x_real)
        feats_fake = self.get_dino_features(x_fake.detach())
        loss = 0
        for fr, ff, head in zip(feats_real, feats_fake, self.disc_heads):
            d_r = head(fr.transpose(1, 2))
            d_f = head(ff.transpose(1, 2))
            loss += F.relu(1 - d_r).mean() + F.relu(1 + d_f).mean()
        return loss / len(self.disc_heads)

    def distill_loss(self, z, x_fake):
        """teacher multi-step ODE output as supervision"""
        with torch.no_grad():
            x_teacher = self.teacher_ode_sample(z, steps=4)
        # pixel-level MSE
        return F.mse_loss(x_fake, x_teacher)

    @torch.no_grad()
    def teacher_ode_sample(self, z, steps=4):
        """teacher runs K-step ODE to generate images as student's distillation target"""
        # ... EDM Heun sampler, implementation omitted
        return self.teacher.sample(z, num_steps=steps)

    def step(self, z, x_real):
        # 1) G output (1-step)
        x_fake = self.G(z)
        # 2) D loss
        self.opt_D.zero_grad()
        loss_D = self.adv_loss_D(x_real, x_fake)
        loss_D.backward()
        self.opt_D.step()
        # 3) G loss (adversarial + distill)
        x_fake = self.G(z)  # recompute (D updated)
        self.opt_G.zero_grad()
        loss_adv = self.adv_loss_G(x_fake)
        loss_dist = self.distill_loss(z, x_fake)
        loss_G = loss_adv + self.lambda_distill * loss_dist
        loss_G.backward()
        self.opt_G.step()
        return loss_G.item(), loss_D.item()

7.7　Code 7: LCM-LoRA attached to SDXL

# Assume a diffusers-style SDXL pipeline is available
from diffusers import StableDiffusionXLPipeline, LCMScheduler
from peft import LoraConfig, get_peft_model

def attach_lcm_lora(sdxl_pipe, lcm_lora_path="latent-consistency/lcm-lora-sdxl"):
    """LCM-LoRA: attach LCM distillation's weight difference as a LoRA.
    In current diffusers (>=0.24) `LCMScheduler`'s teacher-step parameter is `original_inference_steps`
    in the scheduler config; only old community pipelines / early dreamshaper examples use `lcm_origin_steps`.
    """
    # 1) Switch scheduler to LCM-style; teacher-equivalent step count in config
    sdxl_pipe.scheduler = LCMScheduler.from_config(
        sdxl_pipe.scheduler.config,
        original_inference_steps=50,   # current diffusers LCMScheduler API
    )
    # 2) load LCM-LoRA weights
    sdxl_pipe.load_lora_weights(lcm_lora_path)
    # 3) optional: also attach the user's own LoRA (e.g., character LoRA)
    # sdxl_pipe.load_lora_weights("path/to/user_lora", adapter_name="char")
    # sdxl_pipe.set_adapters(["default", "char"], adapter_weights=[1.0, 0.8])
    return sdxl_pipe

# Inference: only 4 steps
pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")
pipe = attach_lcm_lora(pipe)
images = pipe(
    prompt="a cat sitting on a chair",
    num_inference_steps=4,         # key: LCM only needs 4 steps
    guidance_scale=0.0,            # HF LCM-LoRA currently recommends 0.0; 1.0-2.0 also OK
).images

7.8　Code 8: Reflow (Rectified Flow distillation)

@torch.no_grad()
def reflow_generate_pairs(v_net, num_samples, sample_shape, steps=50, device='cuda'):
    """Run current v_θ ODE to generate coupled (x_0, x_1) pairs for reflow retraining.
    sample_shape: tuple, e.g., (D,) for toy data or (C, H, W) for image latents.
    """
    x_0 = torch.randn(num_samples, *sample_shape, device=device)
    x = x_0.clone()
    ts = torch.linspace(0, 1, steps + 1, device=device)
    for i in range(steps):
        t = ts[i].expand(num_samples)
        dt = ts[i + 1] - ts[i]
        x = x + dt * v_net(x, t)
    return x_0, x  # x_1 = ODE(x_0; v_θ), naturally coupled

def reflow_loss(v_net, x_0, x_1, t_dist='uniform'):
    """RF + reflow loss: retrain v_θ^{(k+1)} on coupled (x_0, x_1^{(k)}) pairs"""
    B = x_0.shape[0]
    if t_dist == 'uniform':
        t = torch.rand(B, device=x_0.device)
    else:  # logit-normal (SD3 style)
        t = torch.sigmoid(torch.randn(B, device=x_0.device))

    # Broadcast to any rank: t shape -> (B, 1, 1, ..., 1) aligned with x_0
    t_view = t.view(B, *([1] * (x_0.ndim - 1)))
    x_t = (1 - t_view) * x_0 + t_view * x_1
    # target: u_t = x_1 - x_0 (constant)
    target = x_1 - x_0

    pred = v_net(x_t, t)
    return F.mse_loss(pred, target)

# Full reflow training procedure
def train_with_reflow(v_net, data_loader, num_reflow_rounds=2, device='cuda'):
    """1st round: independent pairs; subsequent rounds: coupled pairs (reflow)"""
    # Round 0: independent pairs (ordinary RF training)
    for batch in data_loader:
        x_1 = batch[0] if isinstance(batch, (tuple, list)) else batch
        x_0 = torch.randn_like(x_1)
        loss = reflow_loss(v_net, x_0, x_1)
        # ... optimizer step

    # Infer sample shape from data_loader (don't rely on custom .dim attribute)
    first_batch = next(iter(data_loader))
    first_x1 = first_batch[0] if isinstance(first_batch, (tuple, list)) else first_batch
    sample_shape = tuple(first_x1.shape[1:])  # e.g., (D,) or (C, H, W)

    # Rounds 1, 2, ...: reflow
    for k in range(num_reflow_rounds):
        # 1) Use current v_net to generate coupled pairs
        x_0_pool, x_1_pool = reflow_generate_pairs(
            v_net, num_samples=10_000, sample_shape=sample_shape, device=device
        )
        # 2) Retrain on coupled pairs
        from torch.utils.data import TensorDataset, DataLoader
        coupled_loader = DataLoader(
            TensorDataset(x_0_pool, x_1_pool), batch_size=64, shuffle=True
        )
        for x_0_b, x_1_b in coupled_loader:
            loss = reflow_loss(v_net, x_0_b, x_1_b)
            # ... optimizer step
    return v_net

§8 Production Landscape

8.1　Mainstream production few-step models list (2024-2026)

Model	Distillation method	Base	Step	Resolution	Open source
LCM-SDXL / LCM-LoRA	LCM (consistency on latent)	SDXL	4-8	1024²	✅
SDXL-Turbo	ADD	SDXL	1	512²	✅ (weights only)
SDXL-Lightning	progressive + GAN	SDXL	1/2/4/8	1024²	✅ LoRA
TCD-SDXL	trajectory CD	SDXL	4-8	1024²	✅
DMD2-SDXL	DMD2 (score gap + GAN)	SDXL	1/4	1024²	✅
SD3-Turbo	LADD	SD3 8B	4	1024²	API only
FLUX.1-schnell	LADD-style	FLUX 12B	1-4	1024²	✅ Apache 2.0
PixArt-LCM / PixArt-α-Lightning	LCM / Lightning	PixArt-α	4-8	1024²	✅
SDXS	feature alignment + GAN	SDXL	1	512²	✅

⚠️ "Open source" ✅ doesn't mean fully commercially usable — SDXL-Turbo had a non-commercial early license; FLUX-schnell Apache 2.0 is commercial-OK but FLUX-pro (teacher) is closed-source. Always check the license before production deployment.

8.2　Video Distillation status

Video diffusion distillation only started in 2024-2025:

• AnimateLCM (Wang et al. 2024): apply LCM to AnimateDiff motion module, 4-step video
• VideoCrafter-LCM / CogVideoX-LCM: similar applications
• Hunyuan-Video-Lightning / Wan-Lightning: Lightning-style + temporal-aware D
• rCM (arXiv:2510.08431): extend sCM to Wan 2.1 14B / Cosmos-Predict2, 1-4 step 5-second video

💡 Difficulty of video distillation — for static images D looks at single frames; for video D must look at temporal coherence — one approach is D taking a video clip as input (3D conv backbone), another is single-frame D + flow-consistency loss combined. Far less engineering experience than image distillation.

8.3　Deployment cheat sheet

Scenario	Recommendation	Reason
Mobile / WebGPU	SDXL-Turbo / FLUX-schnell 1-step	latency < 200 ms
Server batch (API)	DMD2-SDXL 4-step / SD3-Turbo	balance of quality + diversity
Secondary development (character LoRA)	LCM-LoRA / SDXL-Lightning LoRA	doesn't break existing ecosystem
Academic baseline	sCM / CD / iCT	mathematically clean, reproducible
Real-time video	rCM / Wan-Lightning	current SOTA

§9 Failure modes & selection decisions

9.1　Common failure modes

Phenomenon	Possible cause	Countermeasure
Mode collapse (low output diversity)	1-step + pure MSE distill; ADD without GAN	add DMD score gap or GAN loss
Saturated colors (too much red/yellow)	CFG distilled in + too few steps	reduce $w$; use LCM-LoRA 4 step instead of 1 step
High-freq detail blurry	sCM mode-covering	use rCM with mode-seeking regularizer
Text alignment degenerated	one-step CFG distill imprecise	use multi-condition $w$ training (LCM style)
EMA collapse (loss stuck)	EMA decay too high / low	start with 0.9999, monitor spectral norm
Wrong pseudo-Huber c	too small → L1 dominates (non-smooth); too large → degenerates to L2	iCT paper $c = 0.00054$ (CIFAR), $c \propto \sqrt{D}$ (D=dim)
JVP NaN (sCM)	warmup ratio $r$ rises too fast	NCS warmup: $r=0$ for first ~5% steps, then gradually increase

9.2　Selection decision tree

Q1: What's the base model?
  ├─ SD 1.5 / SDXL → LCM-LoRA (ecosystem) or SDXL-Lightning
  ├─ SD3 / FLUX → LADD (official SD3-Turbo / FLUX-schnell)
  ├─ DiT / self-trained → sCM (continuous-time) or DMD2
  └─ Pixel-space (CIFAR/ImageNet) → CD / iCT / EDM teacher

Q2: Target NFE?
  ├─ 1-step → DMD2 / ADD / sCM / iCT
  ├─ 2-4 step → LCM / TCD / LADD
  └─ 8 step OK → progressive distillation / EDM Heun sufficient

Q3: Does it need CFG?
  ├─ Yes (text-to-image) → use a CFG-distill-supporting method (LCM / LADD)
  └─ No (unconditional) → CD / DMD usable directly

Q4: Does it need diversity?
  ├─ High (commercial product) → DMD2 / rCM (with reverse-KL / mode-seeking)
  └─ Low (fixed prompt) → ADD / Lightning suffice

9.3　Evaluation metrics

• FID (Fréchet Inception Distance): standard image quality + diversity metric; lower is better
• CLIP Score / CLIPSim: text-image alignment
• GenEval (SD3): structured evaluation of object counts / colors / positions
• HPSv2 / ImageReward: human preference scores
• PRD / Precision-Recall: measure "fake image quality" and "covering diversity" respectively
• Step-wise FID: not just 1-step but 2/4/8-step too

⚠️ The FID pitfall — FID is insensitive to mode collapse — it only computes mean/cov, potentially missing a student that only generates 50% of modes. Must be paired with Precision-Recall or IS / Coverage metrics for cross-validation.

§10 25 frequently-asked interview questions (L1 must-know · L2 intermediate · L3 top lab)

L1 must-know (potentially asked at any ML / diffusion role)

Q1. Why does diffusion need distillation? Can't you just reduce steps?

• Diffusion sampling has 50-1000 NFE, network forwards >95% of total latency, production wants < 1 s real-time

• Direct step reduction (e.g., 50→4) blows up ODE discretization error: 1-step Euler error is $O(\Delta t)$, at 4 steps $\Delta t$ is 12.5× larger, image high-freq details collapse

• The essence of distillation: retrain a student that learns to "jump from any $x_t$ directly to $x_0$" (CM) or "match teacher's output distribution" (DMD) or "fool D" (ADD)

Saying only "diffusion is slow" without noting that network forward is the bottleneck; thinking DPM-Solver is enough (10-NFE is its physical limit).

Q2. Write the Consistency Models consistency loss.

$\mathcal{L}_\text{CD} = \mathbb{E}\big[d\big(f_\theta(x_{t_{n+1}}, t_{n+1}),\; f_{\theta^-}(\hat x_{t_n}, t_n)\big)\big]$

• $\theta^-$ is the EMA target

• $\hat x_{t_n} = x_{t_{n+1}} - (t_{n+1} - t_n) v_\phi(x_{t_{n+1}}, t_{n+1})$ from one teacher ODE step

• $d$ = L2 or LPIPS

Confusing EMA target with stop-gradient (the former is learnable, the latter is just stop-grad); forgetting boundary needs EDM precond.

Q3. Differences between CD and CT?

• CD (Consistency Distillation): has teacher diffusion, uses it for one-step ODE to compute $\hat x_{t_n}$

• CT (Consistency Training): no teacher, uses $\hat x_{t_n} = x_0 + t_n \epsilon$ (same epsilon, different noise levels)

• Original CT had quality far below CD (CIFAR FID 8.7 vs 3.55); iCT brought CT to 2.83 via pseudo-Huber + lognormal sigma + curriculum, beating CD

Saying only "CT doesn't need teacher" without iCT improvements; thinking CD is always better than CT (counterexample: iCT).

Q4. What is the core idea of DMD?

• Treat student $G_\theta(z) \to x$ as a direct generator

• Optimize reverse-KL: $\text{KL}(p_\text{fake}^\theta \| p_\text{real})$, gradient = score gap × ∂G/∂θ

  $\nabla_\theta \text{KL} = -\mathbb{E}[(s_\text{real} - s_\text{fake}) \cdot \partial G_\theta]$

• $s_\text{real}$ = teacher diffusion (frozen), $s_\text{fake}$ = fake diffusion trained on $G_\theta$'s outputs

Saying only "DMD is distribution matching" without writing the gradient; not knowing $s_\text{fake}$ is also a diffusion model.

Q5. Fundamental difference between DMD and GAN?

• GAN uses discriminator for binary signal (real/fake), low sample efficiency

• DMD uses score gap = ∇log(p_real/p_fake) for dense vector-field signal, telling student where each point should move

• Physical intuition: the score gap is the "force" pushing the student from $p_\text{fake}$ toward $p_\text{real}$

• DMD2 actually adds GAN loss as a fidelity auxiliary

Not knowing the physical meaning of score gap; thinking DMD is just a GAN variant.

Q6. Why does ADD (SDXL-Turbo) use DINOv2 as discriminator?

• Ordinary GAN training D from scratch is unstable for 1-step generators (mode collapse / won't train)

• DINOv2 provides "pretrained high-level perceptual features", anchors the discrimination problem in strong semantic space

• Multiple layer heads + hinge loss for training stability

• Also saves D training cost (D backbone frozen, only 1×1 conv heads trained)

Saying only "DINOv2 is useful" without explaining why not from scratch; not knowing heads are multi-layer.

Q7. Core differences between LCM and CM?

• Space: LCM in VAE latent space (64× compute savings); CM in pixel space

• CFG: LCM feeds guidance scale $w$ as extra condition $f_\theta(x_t, t, c, w)$, no double forward at inference; CM paper doesn't address CFG

• Skipping-step distillation: LCM uses $k$-step skipped teacher for faster convergence

Confusing LCM with LCM-LoRA (the latter packages LCM as a LoRA adapter).

Q8. Rectified Flow's reflow algorithm?

1. Train $v_\theta^{(1)}$ with independent pairs $(x_0, x_1) \sim p_0 \otimes p_\text{data}$

2. Run ODE with $v_\theta^{(1)}$ to generate coupled pairs $(x_0, x_1^{(1)})$

3. Retrain $v_\theta^{(2)}$ on coupled pairs; new trajectory is straighter

• Transport-cost-non-increasing theorem: each reflow does not increase total transport cost

• 1-2 rounds of reflow makes 1-step Euler comparable to 50-step

Saying only "reflow makes trajectories straight" without deriving the transport cost monotonicity; forgetting InstaFlow is reflow applied to SD.

Q9. Method differences between SDXL-Turbo and SD3-Turbo / FLUX-schnell?

• SDXL-Turbo (ADD): DINOv2 pixel-space discriminator + teacher MSE distillation

• SD3-Turbo / FLUX-schnell (LADD): move D to latent space, use teacher MM-DiT intermediate-layer features as D backbone, supports high resolution + high-param base

• ADD is limited by DINOv2 input resolution (≤ 518); LADD has no such limit

• FLUX-schnell is the RF version of LADD

Not knowing LADD is the latent version of ADD; thinking FLUX-schnell is ordinary CM distillation.

Q10. Why is LCM-LoRA's ecosystem value large?

• LCM-trained "weight diff" $\Delta\theta = \theta_\text{LCM} - \theta_\text{SD}$ can be parametrized as LoRA ($r \in [8, 64]$)

• User's existing SD 1.5 / SDXL fine-tunes (DreamShaper / character LoRAs) don't need retraining; just attach LCM-LoRA for 4-step generation

• Ecosystem side: SD has tens of thousands of fine-tunes; LCM-LoRA is the only acceleration solution that doesn't break the existing ecosystem

• Low training cost (~30 A100 hours / SDXL)

Saying only "LCM-LoRA is the LoRA version of LCM" without ecosystem significance; forgetting LCM-LoRA's training cost is much smaller than LCM.

L2 intermediate (research-oriented · need to know diffusion training details)

Q11. What are iCT's four improvements over CT? Why can EMA be dropped?

Four changes:

1. No EMA: directly use stop_grad for target

2. Pseudo-Huber loss: $\sqrt{\|a-b\|^2 + c^2} - c$ replaces LPIPS, adaptively robust

3. Lognormal noise schedule: $\log\sigma \sim \mathcal{N}(P_\text{mean}, P_\text{std}^2)$ replaces uniform

4. Step-count curriculum: $N$ grows from 10 to 1280

Why EMA can be dropped: original CT's EMA prevents "the network's output collapsing to trivial $f \equiv 0$ when differentiating against itself". With pseudo-Huber + lognormal sigma, the loss surface is more "convex" (the small-residual region dominates), and stop_grad alone suffices to prevent collapse.

Memorizing the changes without knowing why; thinking EMA is always required (still a misconception).

Q12. Why can the sCM TrigFlow parametrization simultaneously simplify EDM precond / PF-ODE / CM?

$x_t = \cos(t) x_0 + \sin(t) z,\; t \in [0, \pi/2]$

• EDM precond: $D_\theta = \cos(t) x_t - \sin(t) F_\theta$, boundary automatically satisfied ($D = x_0$ at $t=0$)

• PF-ODE: $dx_t/dt = -\sin(t) x_0 + \cos(t) z$, clean

• CM: consistency function $f_\theta = \cos(t) x_t - \sin(t)(\sigma_d F_\theta)$, same form as EDM

• Key: $\cos^2 + \sin^2 = 1$ (variance preservation), and $d\cos/dt = -\sin$ gives a "natural" ODE term

Saying only "sin cos are simple" without explaining why four things "happen to" simplify; not knowing $\sigma^2 + \alpha^2 = 1$ is the VP condition.

Q13. What is sCM's NCS warmup? Why is it needed?

• NCS = Noise → Consistency → Score (warmup order)

• Early training $r \approx 0$, sCM loss degenerates to standard score matching (learn $F_\theta \approx \epsilon$)

• Gradually increase $r$; consistency term (JVP) takes over

• Without warmup, directly $r = 1$: network hasn't learned the score yet, JVP is a noise direction, training NaN

• Similar to "train D first, then alternate" in GAN training: establish base representation first, then increase difficulty

Saying only warmup is a "training trick" without knowing score precedes consistency; not knowing JVP without convergence leads to NaN.

Q14. What did DMD2 change vs DMD? Why are these changes important?

Three changes:

1. Drop the regression loss: DMD v1 required pre-generated teacher pairs (expensive + mode-limited); DMD2 relies purely on score gap + GAN

2. Add GAN loss: discriminator sees real data + student outputs, providing high-freq detail supervision

3. Multi-step student: at training simulate the $K$-step inference trajectory so the same weights support 1/2/4-step

Importance:

• Drop regression → data scale unlocked (no longer dependent on teacher pairs)
• Add GAN → complementary to DMD score gap (score gives distribution-level signal, GAN gives sample-level fidelity)
• Multi-step → production flexibility (same model can switch 1-step / 4-step)

Not knowing why multi-step is needed ("isn't 1-step enough?"); forgetting GAN in DMD2 is auxiliary rather than main loss.

Q15. Two-stage CFG distillation flow?

Stage 1 - Guidance distillation (Meng 2023): train $\tilde\epsilon_\theta(x, c, w)$, feed $w$ as a condition —

$\mathcal{L}_\text{guide} = \|\tilde\epsilon_\theta(x_t, c, w) - \tilde\epsilon^*(x_t, c, w)\|^2$

where $\tilde\epsilon^* = (1+w) \epsilon_\theta(x, c) - w \epsilon_\theta(x, \emptyset)$ is the teacher's two-forward CFG output. Student does one forward.

Stage 2 - Step distillation: on top of stage 1, stack progressive distillation, compressing 32 → 4/2/1 step. LCM does stage 1 + 2 simultaneously.

Knowing CFG distillation but not writing two stages; not knowing LCM-LoRA's $w$-condition comes from this.

Q16. Role of EDM preconditioning in CM?

$f_\theta(x, \sigma) = c_\text{skip}(\sigma) x + c_\text{out}(\sigma) F_\theta(c_\text{in} x, c_\text{noise})$

Specific values (Song 2023):

c_\text{skip} = \sigma_d^2 / ((\sigma - \sigma_\min)^2 + \sigma_d^2), c_\text{out} = \sigma_d (\sigma - \sigma_\min) / \sqrt{\sigma_d^2 + \sigma^2}

Role:

1. Boundary automatically satisfied: at \sigma = \sigma_\min, $c_\text{skip} = 1, c_\text{out} = 0$, so f(x, \sigma_\min) = x (identity)

2. Unit-variance: makes $F_\theta$ input/output variance independent of $\sigma$, stabilizing training

Not knowing c_\text{skip}(\sigma_\min) = 1 is the key to boundary; confusing EDM precond with score-based reparam.

Q17. If ADD's distillation loss uses pixel MSE rather than score gap, what's the problem?

• Pixel MSE is mode-covering + blurry: student output = teacher mean, losing details

• Need to add GAN loss to compensate for high-freq → ADD must have GAN (unlike DMD which can be pure score gap)

• That's why ADD's distill loss is only an "anchor" (preventing severe mode collapse); the main battle is GAN

• Compare DMD: score gap → dense per-pixel gradient, can generate without GAN (DMD2 adds GAN for further gains)

Saying only "MSE is blurry" without why; thinking ADD doesn't need GAN to work.

Q18. What capabilities does CTM add vs CM?

• CM: only learns $f(x_t, t) \to x_0$ (trajectory endpoint)

• CTM: learns $G(x_t, t, s)$, any $s < t$ can be jumped to

• Practical benefits:

  • Runtime-selectable inference step count (CM is fixed)
  • Controllable intermediate states (suitable for image-to-image / inpainting)
  • Training + score-matching auxiliary loss to prevent triviality

• FID: CIFAR 1-step 1.73 / ImageNet 64 1.92 (SOTA)

Saying only "CTM is the trajectory version of CM" without why "any s" is useful; confusing CTM and TCD (the latter is an LCM improvement).

Q19. How to prove reflow's transport-cost monotonicity?

Setup: consider independent pairs $(x_0, x_1) \sim p_0 \otimes p_1$, initial cost $C^{(0)} = \mathbb{E}\|x_1 - x_0\|^2$.

Reflow: run ODE with $v_\theta^{(1)}$ to get coupled $(x_0, x_1^{(1)})$, cost $C^{(1)} = \mathbb{E}\|x_1^{(1)} - x_0\|^2$.

Key observations:

• $x_1^{(1)} = x_0 + \int_0^1 v_\theta^{(1)}(x_t, t)\, dt$

• In $L^2$, $\|x_1^{(1)} - x_0\| = \|\int v\, dt\| \le \int \|v\|\, dt$ (Cauchy-Schwarz)

• And $v_\theta^{(1)}$'s training objective is to minimize $\mathbb{E}\|v - (x_1 - x_0)\|^2$ → in expectation $\|v\| \approx \|x_1 - x_0\|$

• Rigorous theorem (Liu 2022 Theorem 3.6): $C^{(k+1)} \le C^{(k)}$ (from the OT view, reflow does not increase transport cost)

Intuition: straight lines are OT solutions ⇒ repeated reflow pushes toward the OT solution.

Saying only "trajectories become straight" without writing transport cost; not knowing the Cauchy-Schwarz intuition.

Q20. How do you evaluate the distilled student? Is FID alone enough?

Why FID isn't enough:

• FID only computes Inception feature mean + cov, insensitive to mode collapse (a student generating 50% of modes may still have low FID)

• Insensitive to high-freq details (Inception backbone pools heavily at 224×224)

Necessary auxiliary metrics:

• Precision / Recall (Kynkäänniemi 2019): respectively measure "fake image quality" and "covering diversity"

• CLIP Score: text-image alignment

• HPSv2 / ImageReward / PickScore: human preference

• Step-wise FID: look at 1/2/4/8-step, avoid only optimizing for 1-step

• Mode count / coverage: directly count how many real clusters the generation covers

Saying only "FID is enough"; forgetting human-preference evaluation is mandatory for production deployment.

L3 top-lab questions (research depth · derivation required)

Q21. Derive the continuous-time form of the consistency loss from the PF-ODE.

PF-ODE: $dx_t/dt = v_\phi(x_t, t)$ (teacher).

Consistency definition: $f_\theta(x_{t+\Delta t}, t+\Delta t) = f_\theta(x_t, t)$ along the same ODE trajectory.

First-order Taylor:

$f_\theta(x_{t+\Delta t}, t+\Delta t) = f_\theta(x_t, t) + \Delta t \cdot \frac{d f_\theta}{dt} + O(\Delta t^2)$

where $\frac{d f_\theta}{dt} = \partial_t f_\theta + (\nabla_x f_\theta)^\top \cdot \dot x_t = \partial_t f_\theta + (\nabla_x f_\theta)^\top v_\phi$ (chain rule + PF-ODE substitution).

Continuous-time consistency loss:

$\mathcal{L}_\text{cont}(\theta) = \mathbb{E}\!\left[\Big\|\partial_t f_\theta(x_t, t) + \nabla_x f_\theta(x_t, t) \cdot v_\phi(x_t, t)\Big\|^2\right]$

Discretization (original CM): use $\hat x_{t_n} = x_{t_{n+1}} + (t_n - t_{n+1}) v_\phi(\cdot)$ as teacher Euler, and $f_{\theta^-}$ as target —

$\mathcal{L}_\text{CD} \approx \mathbb{E}\|f_\theta(x_{t_{n+1}}, t_{n+1}) - f_{\theta^-}(\hat x_{t_n}, t_n)\|^2$

Knowing only the discrete loss without deriving the continuous form; confusing $\partial_t$ and $d/dt$ (the former is partial, the latter is total).

Q22. Physical meaning of DMD's two scores? Why must we use fake score and not zero?

Physical meaning:

• $s_\text{real}(x, t) = \nabla_x \log p_\text{real}(x_t)$: the "force" pushing $x_t$ toward real data

• $s_\text{fake}(x, t) = \nabla_x \log p_\text{fake}(x_t)$: score of student's current output distribution

• Difference $s_\text{real} - s_\text{fake} = \nabla_x \log(p_\text{real}/p_\text{fake})$: reverse-KL gradient direction

Why fake score is necessary:

• If only $s_\text{real}$ (i.e., $s_\text{fake} \equiv 0$): equivalent to pushing student toward the "modes of $p_\text{real}$" — mode collapse

• $s_\text{fake}$ provides the signal "no need to push further to already-covered positions", similar to GAN's D providing contrastive feedback

• Mathematically: $\mathbb{E}_{p_\text{fake}}[s_\text{real} - s_\text{fake}]$ is a Stein discrepancy, the correct distribution-matching signal

Implementation:

• $s_\text{real}$ = teacher diffusion (frozen)
• $s_\text{fake}$ = a small diffusion model, trained with DSM on $G_\theta$'s current outputs, jointly trained with $G_\theta$

Saying only "DMD uses score" without distinguishing the two roles; not knowing $s_\text{fake}$ needs joint training.

Q23. What's the essence of the scale difference between ADD and LADD? Why can't ADD scale to SD3 8B / FLUX 12B?

ADD bottlenecks:

1. DINOv2 input resolution: ADD uses DINOv2 base (518²) as D backbone; above this resolution requires patching / downsampling, limiting 1024² input

2. Pixel-space distill: MSE(G(z), teacher_ode(z)) requires VAE decode, back-prop through VAE is expensive and unstable

3. Discriminator capacity: DINOv2 ViT-L 1B params is far less than SD3 8B / FLUX 12B's base, D expressiveness insufficient

LADD's solutions:

1. Latent space: D directly on VAE latent (128×128×16 for SD3), resolution-agnostic

2. Teacher's own MM-DiT blocks as D backbone: extract SD3's transformer blocks and fine-tune as D, capacity automatically matches base scale

3. Score distill in latent: avoids VAE back-prop

Results: SDXL-Turbo (2.6B SDXL ADD) at 1024² is the ADD limit; SD3-Turbo (8B LADD) / FLUX-schnell (12B LADD-style) need LADD for stable training.

Saying only "LADD is in latent space" without why ADD can't scale; forgetting DINOv2 resolution is a hard cap.

Q24. Mathematical relationship between Flow-OPD (2026 arXiv:2605.08063) and DMD?

📍 Clarification: Flow-OPD is primarily a multi-reward RL alignment paper, somewhat tangential to this tutorial's few-step inference distillation thread; it appears here because the name contains "Distillation". See diffusion_post_training_tutorial.md for detailed discussion.

DMD: reverse-KL gradient (on student output distribution), single teacher, single objective = match teacher distribution.

Flow-OPD: on-policy distillation with multiple reward-specific teachers (each GRPO fine-tuned as a specialist on one reward); it is an alignment paper (multi-reward alignment), not an inference distillation paper.

Saying "DMD reduces to OPD" is wrong: DMD's reverse-KL and OPD's multi-teacher vector-field weighting have different mathematical objectives — one is distribution matching, the other is reward-aware policy supervision. Their priorities differ (single-teacher distribution match vs multi-reward alignment) with no reduction relationship. In interviews don't say "DMD is a special case of OPD" or vice versa — no reliable mathematical basis.

Practical significance (safer phrasing):

• DMD is more suited to few-step inference (single objective: match teacher distribution)
• Flow-OPD is more suited to multi-reward alignment (multi-reward alignment + on-policy training)
• They solve different problems and are not substitutes; for detailed alignment content see diffusion_post_training_tutorial.md

Treating Flow-OPD as just "another inference distillation" is a confusion — its multi-reward / RL nature is central; similarly, don't claim "reduction to DMD" as established math.

Q25. Design a distillation scheme that runs 1024² video in 4 steps + preserves temporal coherence. Give the loss and D design.

Setup:

• Teacher: 50-step video diffusion (e.g., Wan 2.1 14B, Rectified Flow)
• Student: 4-step video generator $G_\theta(z_{1:T}, c)$
• Target: 1024² × 5 sec

Loss combination (rCM-style + LADD-style):

$\mathcal{L}_\text{total} = \underbrace{\mathcal{L}_\text{sCM}^\text{trig}}_{\text{video CM, JVP-based}} + \lambda_1 \cdot \underbrace{\mathcal{L}_\text{score-reg}}_{\text{mode-seeking via score gap}} + \lambda_2 \cdot \underbrace{\mathcal{L}_\text{adv}^\text{video}}_{\text{temporal D}}$

Video Discriminator design:

• Backbone: teacher's own 3D MM-DiT block (latent space, avoid VAE decode)

• Two heads:

  • Spatial head: single-frame latent → real/fake signal (image quality)
  • Temporal head: consecutive $k$-frame latent stack → real/fake (motion realism)

• Optical flow consistency loss (auxiliary): $\mathcal{L}_\text{flow} = \mathbb{E}\|f_\text{flow}(\hat x_{t}, \hat x_{t+1}) - f_\text{flow}(x_t^\text{real}, x_{t+1}^\text{real})\|$

Training tricks:

• Multi-stage: pretrain on static images ($T = 1$) → add temporal D → fine-tune full video
• Curriculum on T: train short clips first ($T = 8$ frame) → longer clips ($T = 80$ frame)
• EMA on G: prevent student outputs from drifting across steps

Evaluation:

• VBench (static quality + dynamic quality across 16 dimensions)
• FVD (Fréchet Video Distance)
• Human comparisons (rCM-style)

Compared baseline: rCM is already approaching this on Wan 2.1 14B — production-grade direction, still developing fast in 2026.

Need to fuse "image distillation + temporal supervision + large base"; using a single D on single frames causes motion collapse; using only score-gap without GAN gives blurry details.

§A Appendix: References

Consistency Models family:

• Song et al. 2023, "Consistency Models", ICML 2023, arXiv:2303.01469
• Song & Dhariwal 2023, "Improved Techniques for Training Consistency Models" (iCT), arXiv:2310.14189
• Lu & Song 2024, "Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models" (sCM / TrigFlow), ICLR 2025, arXiv:2410.11081
• Kim et al. 2023, "Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion" (CTM), ICLR 2024, arXiv:2310.02279
• Luo et al. 2023, "Latent Consistency Models" (LCM), arXiv:2310.04378
• Luo et al. 2023, "LCM-LoRA: A Universal Stable-Diffusion Acceleration Module", arXiv:2311.05556
• Zheng et al. 2024, "Trajectory Consistency Distillation" (TCD), arXiv:2402.19159
• "Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency" (rCM), arXiv:2510.08431 (rCM acronym verified)

Distribution Matching Distillation:

• Yin et al. 2024, "One-step Diffusion with Distribution Matching Distillation" (DMD), CVPR 2024, arXiv:2311.18828
• Yin et al. 2024, "Improved Distribution Matching Distillation for Fast Image Synthesis" (DMD2), NeurIPS 2024, arXiv:2405.14867

Adversarial Distillation:

• Sauer et al. 2023, "Adversarial Diffusion Distillation" (ADD / SDXL-Turbo), arXiv:2311.17042
• Sauer et al. 2024, "Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation" (LADD / SD3-Turbo), arXiv:2403.12015
• Lin et al. 2024, "SDXL-Lightning: Progressive Adversarial Diffusion Distillation", arXiv:2402.13929

Flow / Rectified Flow:

• Liu, Gong & Liu 2022, "Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow", ICLR 2023, arXiv:2209.03003
• Liu et al. 2023, "InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation", ICLR 2024, arXiv:2309.06380
• "Flow-OPD: On-Policy Distillation for Flow Matching Models", arXiv:2605.08063 (Flow-OPD is primarily a multi-reward RL alignment paper, somewhat tangential to this tutorial's few-step inference distillation thread; see diffusion_post_training_tutorial.md)

CFG / Step Distillation:

• Meng et al. 2023, "On Distillation of Guided Diffusion Models", CVPR 2023, arXiv:2210.03142
• Salimans & Ho 2022, "Progressive Distillation for Fast Sampling of Diffusion Models", ICLR 2022, arXiv:2202.00512

Foundations:

• Ho, Jain & Abbeel 2020, "Denoising Diffusion Probabilistic Models", NeurIPS 2020 (DDPM)
• Song et al. 2021, "Score-Based Generative Modeling through Stochastic Differential Equations", ICLR 2021
• Karras et al. 2022, "Elucidating the Design Space of Diffusion-Based Generative Models" (EDM), NeurIPS 2022, arXiv:2206.00364
• Lipman et al. 2023, "Flow Matching for Generative Modeling", ICLR 2023

Production models:

• Stable Diffusion XL: Podell et al. 2024 ICLR
• Stable Diffusion 3: Esser et al. 2024 ICML
• FLUX.1: Black Forest Labs 2024 (technical report)

Diffusion / Flow Distillation Cheat Sheet · main references: Song 2023 (CM), Lu-Song 2024 (sCM), Yin 2024 (DMD/DMD2), Sauer 2023/2024 (ADD/LADD), Liu 2022 (RF)