Agents fail open-ended research tasks in shadow evaluations

Agents tasked with replicating unpublished research papers over six days using thousands of dollars in compute failed to produce viable results. Authors of the original research unambiguously rejected the agent-generated papers, noting that while agents handle engineering tasks well, they struggle with hypothesis selection and navigating failing approaches.