Identifying Outlier Tokens in Diffusion Transformers

This research demonstrates that Diffusion Transformers (DiTs) and pretrained ViT encoders produce high-norm outlier tokens that disproportionately attract attention. These tokens often carry limited local information but impact the generative process in intermediate layers.