Using Model Transcripts to Evaluate Capabilities Beyond Benchmark Saturation

As traditional benchmarks reach saturation, analyzing full solution transcripts provides a way to differentiate model capabilities. Testing shows that protocols extracting information from correctly solved transcripts can predict performance on held-out, unsaturated items.