LLM Forecast Accuracy for AI Agent Freelance Job Success
November 5, 2026
The PAYWORTHY study reveals that LLM judges struggle to predict agent success on freelance tasks unless specific agent constraints are provided. When agents are described generically, model AUC dropped to 0.45–0.56, performing near random chance.
HOW THIS AFFECTS YOU
●
builderYou must provide explicit capability constraints in prompts to ensure LLM judges can accurately evaluate agent performance.
●
founderYou should be cautious when using LLMs to gate-keep or select agents for automated marketplaces.