Surrogate Feedback Enables Recursive Language Model Improvement

To bypass the high cost of evaluating LLM-proposed research via full training runs, this method utilizes cheaper surrogate feedback for reinforcement learning. The approach addresses the risk of policy optimization favoring overestimated proposals by managing the shift in proposal distributions.