A systematic audit of the POPE benchmark reveals that VLM hallucination scores are inflated by improper logit readout methods. The evaluation code uses a two-token lookup that fails to match space-prefixed versions of 'yes' and 'no' generated by SentencePiece, causing significant F1 score discrepancies.
HOW THIS AFFECTS YOU
●
builderBe cautious when comparing your model's hallucination metrics against existing literature.
●
researcherYou should review your VLM evaluation scripts for tokenization-induced biases in readout logic.