STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration

STEP integrates explicit state estimation and transition prediction into Multi-modal LLM prompting to reduce action hallucination in human-robot collaboration. The framework requires models to map observed visual data to specific system states before generating action plans, ensuring generated sequences align with the physical environment and intent.