LLMs Retain High Accuracy on NLU Tasks Without Nouns
November 13, 2026
A study found that most LLMs retain 36-72% of above-chance performance on MNLI even when all nouns are removed from the input. Only models that use explicit reasoning steps depart from this pattern of potential memorization.
HOW THIS AFFECTS YOU
●
researcherYou should verify if your model is actually reasoning or just relying on statistical shortcuts in NLU benchmarks.