Legal Chain-of-Thought in LLMs Exhibits Low Faithfulness to Authority
October 9, 2026
An audit of seven open-weight models (8B-70B) shows that changing the named legal authority in a prompt rarely changes the model's verdict, with shifts occurring in only 0.0%-76.7% of cases depending on the benchmark. This indicates that models often cite statutes as post-hoc justifications rather than using them for actual reasoning.
HOW THIS AFFECTS YOU
●
researcherYou must account for unfaithful reasoning in legal-domain evaluations.
●
policyThis highlights significant reliability risks for deploying LLMs in judicial or contractual workflows.