CMA-ES and TPE Still Beat LLM Agents at Hyperparameter Optimization

Classical HPO algorithms CMA-ES and TPE consistently outperform LLM-based agents including frontier models like Claude Opus when tuning a small language model under fixed compute budgets with a defined search space. Allowing LLMs to edit source code directly narrows but does not close the gap, with OOM failure avoidance being the key differentiator for classical methods.