What Does a Discrete Diffusion Model Learn?
A new paper rigorously proves that discrete diffusion models can be interpreted as denoisers, score ratios, or bridge predictors depending on the coordinate system used. The findings challenge prior assumptions about their training and sampling processes.
- Discrete diffusion models can be interpreted as denoisers, score ratios, or bridge predictors depending on the coordinate system used.
- The Oracle Distance theorem proves the negative ELBO is exactly equal to data entropy plus path KL divergence, not just a bound.
- Prior assumptions about training and sampling processes in discrete diffusion models may need revision.
- The choice of coordinate system affects the generative process, challenging existing methodologies.
Researchers have published a paper that redefines how discrete diffusion models are understood in machine learning. The work starts with a rigorous derivation of the continuous-time Markov chain (CTMC) evidence lower bound (ELBO) for any noising process, including boundary terms that were previously overlooked. The paper then introduces the Oracle Distance theorem, which proves that the negative ELBO is exactly equal to the data entropy plus the path KL divergence between the oracle reverse process and the learned one. This is not just a bound but an exact equality, fundamentally altering the interpretation of what these models learn during training and sampling.
The authors argue that discrete diffusion models can be viewed as denoisers, score ratios, or bridge plug-in predictors depending on the coordinate system used to read the neural network. This insight suggests that the process being trained and sampled can change if the model is interpreted in the wrong coordinate system. The findings have implications for how these models are designed, trained, and evaluated, particularly in scenarios where precise control over the generative process is critical.
Source: What Does a Discrete Diffusion Model Learn?. Read the full piece at the source.
Provides a clearer framework for designing and training discrete diffusion models, improving model reliability.
Introduces foundational concepts in diffusion models with precise mathematical derivations.
- CTMC ELBO
- Continuous-time Markov chain evidence lower bound, a key metric in training diffusion models.
- Oracle Distance theorem
- A theorem proving the negative ELBO equals data entropy plus path KL divergence in discrete diffusion models.
- Path KL divergence
- Kullback-Leibler divergence between probability distributions over paths, used to measure differences in generative processes.
Mantle cell lymphoma artificial intelligence prognostic index using hematoxylin and eosin histology - Nature
MIRA: Multiplayer Interactive World Models trained on Rocket League [R]
