Existence of Prefix-Tuning Solutions for Arbitrary Transformer Outputs

This work proves that for any input-output pair where the output is within the pretrained Transformer's range, a task-specific prefix exists and can be computed via linear systems. This provides a theoretical foundation for the expressiveness of prefix-tuning as a parameter-efficient fine-tuning method.