Limits of Confidence in Diffusion

Discrete diffusion methods like remasking fail to match training distributions when writing multiple tokens per step due to inherent inter-token dependencies. On the ScanAndAdd benchmark, the authors demonstrate that per-position marginal distributions cannot capture joint dependencies, preventing a product of independent distributions from representing a dependent group.