Quantifying Optimistic Bias in Few-Shot Learning Evaluations

Analysis of few-shot learning protocols reveals a 9.66-point optimistic bias caused by domain overlap in pre-training data. The study demonstrates that in-domain pre-training improves performance by 33.41 percentage points compared to no pre-training, suggesting current benchmarks overstate true few-shot capabilities.