Evaluating reliability of downstream fine-tuning in federated pre-training

This study assesses whether pre-training test perplexity accurately predicts downstream performance in federated learning environments. Using a 16M parameter transformer, the researchers compare evaluation protocols to determine which most reliably preserves reference rankings across distributed data.