Discrete Action Spaces Enable GRPO Convergence in Small Models
July 27, 2026
Research shows that vanilla GRPO fine-tuning of a Qwen-0.5B model for continuous quadrotor control leads to entropy collapse and zero success. Switching the action interface to a 5-way categorical choice over PID presets enables successful convergence with up to 100% success rates.
HOW THIS AFFECTS YOU
●
researcherYou should use discrete action spaces instead of continuous control when fine-tuning extremely small models with GRPO.