Open-Book Scoring Overestimates Knowledge in Document-Level LoRA Adapters

Testing on Llama-3.1-8B-Instruct reveals that open-book scores primarily measure a model's reading ability rather than stored knowledge in LoRA adapters. While specialists showed a 0.094 increase in cloze recall, random adapters recovered 82% of the open-book advantage, suggesting the metric fails to distinguish between retrieval and parameter-encoded knowledge.