This method uses executable programs instead of free-text chain-of-thought to explain LLM behavior across datasets. A Consistency metric measures how often the generated program reproduces model outputs, allowing for automatic verification of explanations across multiple inputs.
HOW THIS AFFECTS YOU
●
builderYou can use these programs to automatically check if an LLM's reasoning holds up across a whole dataset.
●
researcherYou can move beyond unreliable text-based reasoning traces to verifiable code-based logic.