Audit Reveals Metadata Leakage in Fake News Benchmarks
September 23, 2026
An audit of the ISOT/Kaggle Fake News corpus shows that classifiers achieve near-perfect F1 scores by exploiting subject metadata and newswire source tags rather than content. Removing these leakage channels and duplicate documents reduces F1 from 0.99 to 0.98, showing residual signal is driven by editorial style.
HOW THIS AFFECTS YOU
●
researcherBe cautious of high accuracy scores on this corpus due to inherent data leakage.