●researcherYou should evaluate your mechanistic observers based on the loss caused by the resulting intervention, not just prediction accuracy.
●policyThis highlights why mechanistic safety monitors may fail to prevent harms even if they accurately identify internal features.