Batching Strategies Impact Test-Time Scaling Efficiency for Small LLMs

Increasing candidate count (N) from 1 to 8 improves GSM8K accuracy by 8.4% for Phi-3-mini and 18.4% for Qwen2.5-1.5B. The research highlights that total candidate budget alone fails to account for the computational costs and latency variations between single batched generation calls versus multiple sequential calls.