ConCI Benchmark Tests Contextual Integrity via Counterfactual Scenarios
October 20, 2026
ConCI is a controlled-counterfactual benchmark comprising 1,592 flows designed to measure how LLMs handle personal information based on context. By using triplets that modify single parameters, it distinguishes true contextual sensitivity from simple general refusal tendencies.
HOW THIS AFFECTS YOU
●
researcherYou can use these triplets to isolate specific failure modes in model contextual reasoning.
●
policyThis provides a more rigorous way to evaluate whether models respect privacy context or just trigger safety refusals.