Syntax-Informed Positional Embeddings (SiPE) for Transformers
August 7, 2026
SiPE injects learned syntactic priors from dependency parses into existing positional embedding families without altering self-attention. For autoregressive decoders using relative PE, multiplicative coupling with the relative-position term yields the strongest performance gains.
HOW THIS AFFECTS YOU
●
researcherYou can improve transformer performance by integrating syntactic structure into positional encoding.