Abstract: While commonsense knowledge acquisition and reasoning has traditionally been a core research topic in the knowledge representation and reasoning community, recent years have seen a surge of interest in the natural language processing community in developing pre-trained models and testing their ability to address a variety of newly designed commonsense knowledge reasoning and generation tasks. This paper presents a survey of these tasks, discusses the strengths and weaknesses of state-of-the-art pre-trained models for commonsense reasoning and generation as revealed by these tasks, and reflects on future research directions.
Introduction
In the quest for Artificial General Intelligence, "commonsense" remains a significant hurdle. It encompasses the practical, everyday knowledge about how the world works—such as knowing that a dropped glass might break—which humans rarely explicitly document. Historically, researchers attempted to manually build massive knowledge bases like Cyc. Recently, the field has shifted toward leveraging Pre-trained Language Models (PLMs). This survey investigates whether these models truly possess commonsense or are merely "stochastic parrots" exploiting statistical patterns.
Probing and Enhancement Methods
To evaluate and improve the commonsense capabilities of PLMs like BERT and GPT-3, researchers employ several key strategies:
- Probing (Cloze Tests): Relational facts are converted into "fill-in-the-blank" templates. For instance, a model might be asked to complete the sentence "Dante was born in [MASK]." Success in these tasks suggests the model has captured specific relational knowledge during pre-training.
- Knowledge Graph Integration: Models are enhanced by injecting structured data from external sources like ConceptNet or ATOMIC, providing a scaffold for more coherent reasoning.
- Fine-tuning and Iterative Refinement: General-purpose models are adapted to specific benchmarks or use multi-pass decoding to ensure logical consistency in generated text.
Benchmarking Commonsense
The survey highlights a diverse array of benchmarks designed to test different facets of commonsense:
- Physical & Linguistic: PIQA focuses on physical affordances, while Winogrande tests pronoun resolution and logic.
- Temporal & Social: MC-TACO evaluates understanding of event ordering, and SocialIQA probes reasoning about emotional states.
- Multimodal Reasoning: Tasks like Visual Commonsense Reasoning (VCR) require models to reason about social interactions and physical contexts within images.
Key Insights and Performance Gaps
While PLMs achieve near-human scores on some benchmarks, their performance is often fragile:
- The "Reporting Bias" Trap: Because humans don't write down the obvious (e.g., "I am breathing"), models may fail to learn fundamental truths, instead picking up stereotypical associations.
- Fragility to Negation: Models often struggle with logical operators. For example, BERT may fail to distinguish between "Birds can fly" and "Birds cannot fly," focusing instead on the keywords.
- Knowledge vs. Reasoning: PLMs excel at fact recall but struggle with novel reasoning, such as identifying unconventional uses for everyday objects.
- The Benchmark Paradox: High scores (e.g., 93.8% on HellaSwag) can be misleading. Models often exploit "annotation artifacts"—predictable patterns in question phrasing—rather than performing true reasoning.
The performance gap is most evident in multimodal tasks. While models approach human levels on text-only benchmarks like HellaSwag (93.85% vs. 95.6% human), they lag significantly on VCR, scoring only 63.15% compared to a human baseline of 85.0%.
Future Directions
The authors identify several critical frontiers for future research:
- Adversarial Filtering: Developing benchmarks that prevent models from "cheating" via lexical overlaps.
- Multimodal Training: Incorporating video and sensor data to help models understand physical "affordances" through interaction.
- Multilinguality: Expanding research beyond English to account for the cultural dependencies of commonsense.
- Graph Densification: Improving the coverage and contextualization of existing knowledge graphs.
Summary updated