Researchers at the Hong Kong University of Science and Technology (HKUST) recently addressed a foundational but often overlooked question in artificial intelligence: What exactly constitutes "commonsense knowledge," and are our current benchmarks accurately measuring it? Their findings suggest that many popular datasets are heavily "polluted" with encyclopedic facts, leading to an overestimation of Large Language Model (LLM) reasoning capabilities.
The Problem of "Blurry" Commonsense
In Natural Language Processing (NLP), commonsense is frequently treated as a catch-all term for any knowledge a human possesses. However, this lack of precision creates significant issues. For instance, knowing the boiling point of water or the size of an electron is an encyclopedic fact (referenced knowledge), not commonsense.
The authors argue that this conceptual blurriness leads to two major problems:
- Dataset Contamination: Benchmarks are filled with facts that can be solved via simple memory retrieval rather than genuine reasoning.
- Overestimated Ability: Because LLMs excel at retrieving memorized facts, high scores on these benchmarks mask their actual struggles with implicit, everyday reasoning.
A Unified Framework for Definition
To resolve this, the researchers grounded their study in Murphy’s (2004) three frameworks for defining concepts: Ideal (general descriptions), Feature (lists of typical traits), and Exemplar (specific examples). They developed a consolidated definition and a set of eight binary features to distinguish "Commonsense" (CS) from "Referenced Knowledge" (R).
Key features used to categorize knowledge include:
- Acquisition (
a_self): Is the knowledge obtained through personal experience and observation rather than formal education? - Content (
cr_social): Does it regard daily human interaction, social etiquette, or the usage of human-created objects? - Evaluation (
sce_heur): Is the knowledge based on mutual subjective belief rather than logical, empirical, or clinical certification?
Statistical analysis using OLS regression and decision trees revealed that a_self (personal experience) and sce_heur (subjective observation) are the strongest predictors of true commonsense. Essentially, if a fact is learned in a classroom or found in a textbook, it is likely referenced knowledge, not commonsense.
Benchmark Pollution and the Performance Gap
The study applied these criteria to popular benchmarks like CommonsenseQA (CSQA) and CommonsenseQA 2.0. The results were striking: 27% of CommonsenseQA and a staggering 56% of CommonsenseQA 2.0 do not actually concern commonsense knowledge. Many instances were found to be specialized engineering or expert knowledge rather than the "implicit mutual belief" that characterizes true commonsense.
When LLMs were tested specifically on the true commonsense subsets versus the referenced knowledge subsets, a clear performance gap emerged. Models like Gemini-Pro, LLaMa2, and Mixtral showed a significant performance drop—between 4 to 7 percentage points—when faced with genuine commonsense reasoning.
Interestingly, ChatGPT exhibited a smaller gap. The authors hypothesize that OpenAI's use of massive conversational datasets for training may better capture "verbalized commonsense"—the kind of social etiquette and implicit rules that rarely appear in formal literature but are common in human dialogue.
Conclusion and Future Directions
The study highlights a critical "reporting bias" in AI training: true commonsense is rarely written down because it is considered too obvious (e.g., "you must be awake to eat"). This makes it inherently harder for models to learn from text alone compared to documented facts.
While acknowledging the subjectivity of commonsense across cultures, the authors call for the NLP community to "de-blur" these boundaries. By building more rigorous benchmarks that isolate genuine commonsense from simple fact retrieval, researchers can better evaluate and improve the reasoning capabilities of future AI systems.
Summary updated