Model Organisms Are Leaky: Perplexity Differencing Often Reveals Finetuning Objectives

Researchers at arXiv (2605.00994) demonstrate that perplexity differencing, ranking completions by the perplexity gap between reference and finetuned models using random short prefills, reliably surfaces hidden finetuning objectives from model organisms across 76 models ranging 0.5B to 70B parameters, requiring no model internals or behavioral priors. This black-box auditing method exploits overgeneralization, where finetuned models leak target behaviors outside intended contexts. Red teamers, safety auditors, and alignment researchers should prioritize this over existing interpretability-heavy approaches for scalable behavioral fingerprinting.