MMDiff Extracts Multi-Modal Outputs from Frozen Diffusion Transformers

MMDiff adds lightweight decoder heads to a frozen diffusion transformer to jointly generate images and dense perceptual outputs like semantic segmentation, using multi-timestep feature fusion along the denoising trajectory. Multi-timestep fusion with spatially varying aggregation weights improves segmentation by up to 28.7% mIoU over single-timestep extraction.