What Really is Commonsense Knowledge?

Quyet V. Do, Junze Li, Tung-Duong Vuong, Zhaowei Wang, Yangqiu Song, Xiaojuan MaarXiv 2024 · DOI · arXiv

Researchers at the Hong Kong University of Science and Technology (HKUST) recently addressed a foundational but often overlooked question in artificial intelligence: What exactly constitutes "commonsense knowledge," and are our current benchmarks accurately measuring it? Their findings suggest that many popular datasets are heavily "polluted" with encyclopedic facts, leading to an overestimation of Large Language Model (LLM) reasoning capabilities.

The Problem of "Blurry" Commonsense

In Natural Language Processing (NLP), commonsense is frequently treated as a catch-all term for any knowledge a human possesses. However, this lack of precision creates significant issues. For instance, knowing the boiling point of water or the size of an electron is an encyclopedic fact (referenced knowledge), not commonsense.

The authors argue that this conceptual blurriness leads to two major problems:

  1. Dataset Contamination: Benchmarks are filled with facts that can be solved via simple memory retrieval rather than genuine reasoning.
  2. Overestimated Ability: Because LLMs excel at retrieving memorized facts, high scores on these benchmarks mask their actual struggles with implicit, everyday reasoning.

A Unified Framework for Definition

To resolve this, the researchers grounded their study in Murphy’s (2004) three frameworks for defining concepts: Ideal (general descriptions), Feature (lists of typical traits), and Exemplar (specific examples). They developed a consolidated definition and a set of eight binary features to distinguish "Commonsense" (CS) from "Referenced Knowledge" (R).

Key features used to categorize knowledge include:

  • Acquisition (a_self): Is the knowledge obtained through personal experience and observation rather than formal education?
  • Content (cr_social): Does it regard daily human interaction, social etiquette, or the usage of human-created objects?
  • Evaluation (sce_heur): Is the knowledge based on mutual subjective belief rather than logical, empirical, or clinical certification?
Table 1: Taxonomy of features used to distinguish Commonsense (CS) from Referenced (R) knowledge.

Statistical analysis using OLS regression and decision trees revealed that a_self (personal experience) and sce_heur (subjective observation) are the strongest predictors of true commonsense. Essentially, if a fact is learned in a classroom or found in a textbook, it is likely referenced knowledge, not commonsense.

Benchmark Pollution and the Performance Gap

The study applied these criteria to popular benchmarks like CommonsenseQA (CSQA) and CommonsenseQA 2.0. The results were striking: 27% of CommonsenseQA and a staggering 56% of CommonsenseQA 2.0 do not actually concern commonsense knowledge. Many instances were found to be specialized engineering or expert knowledge rather than the "implicit mutual belief" that characterizes true commonsense.

When LLMs were tested specifically on the true commonsense subsets versus the referenced knowledge subsets, a clear performance gap emerged. Models like Gemini-Pro, LLaMa2, and Mixtral showed a significant performance drop—between 4 to 7 percentage points—when faced with genuine commonsense reasoning.

Table 2: Performance comparison of various LLMs on Commonsense vs. Referenced subsets of popular benchmarks.

Interestingly, ChatGPT exhibited a smaller gap. The authors hypothesize that OpenAI's use of massive conversational datasets for training may better capture "verbalized commonsense"—the kind of social etiquette and implicit rules that rarely appear in formal literature but are common in human dialogue.

Conclusion and Future Directions

The study highlights a critical "reporting bias" in AI training: true commonsense is rarely written down because it is considered too obvious (e.g., "you must be awake to eat"). This makes it inherently harder for models to learn from text alone compared to documented facts.

While acknowledging the subjectivity of commonsense across cultures, the authors call for the NLP community to "de-blur" these boundaries. By building more rigorous benchmarks that isolate genuine commonsense from simple fact retrieval, researchers can better evaluate and improve the reasoning capabilities of future AI systems.

Summary updated

Abstract

Commonsense datasets have been well developed in Natural Language Processing, mainly through crowdsource human annotation. However, there are debates on the genuineness of commonsense reasoning benchmarks. In specific, a significant portion of instances in some commonsense benchmarks do not concern commonsense knowledge. That problem would undermine the measurement of the true commonsense reasoning ability of evaluated models. It is also suggested that the problem originated from a blurry concept of commonsense knowledge, as distinguished from other types of knowledge. To demystify all of the above claims, in this study, we survey existing definitions of commonsense knowledge, ground into the three frameworks for defining concepts, and consolidate them into a multi-framework unified definition of commonsense knowledge (so-called consolidated definition). We then use the consolidated definition for annotations and experiments on the CommonsenseQA and CommonsenseQA 2.0 datasets to examine the above claims. Our study shows that there exists a large portion of non-commonsense-knowledge instances in the two datasets, and a large performance gap on these two subsets where Large Language Models (LLMs) perform worse on commonsense-knowledge instances.
BibTeX
@misc{do2024what,
  author = {Quyet V. Do and Junze Li and Tung-Duong Vuong and Zhaowei Wang and Yangqiu Song and Xiaojuan Ma},
  title = {{What Really is Commonsense Knowledge?}},
  year = {2024},
  eprint = {2411.03964},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.03964},
}
Figure