Commonsense Reasoning Using WordNet and SUMO: a Detailed Analysis

Javier Álvez, Itziar Gonzalez-Dios, German Rigau

Abstract

This paper presents a rigorous manual audit of a large-scale benchmark designed to evaluate artificial commonsense reasoning. By examining the mapping between the WordNet lexical database and the Suggested Upper Merged Ontology (SUMO), the authors identify critical bottlenecks in current logical models. The analysis reveals that while automated theorem proving is sound, the semantic bridge between natural language and formal logic—particularly for adjectives and complex hierarchies—remains a significant challenge for modern AI systems.

Introduction

Developing artificial intelligence with "commonsense" often involves linking lexical resources like WordNet, which organizes human language, with formal ontologies like SUMO, which provides logical rules about the world. This mapping allows words to be treated as logical concepts, enabling Automated Theorem Provers (ATPs) to reason about everyday scenarios. However, failures in these systems are often opaque. This research addresses the lack of transparency by manually inspecting a representative sample of reasoning problems to diagnose whether failures stem from mapping errors, ontological gaps, or theorem prover limitations.

Methodology and Datasets

The study utilizes Adimen-SUMO, a first-order logic translation of SUMO, to test "Competency Questions" (CQs) derived from WordNet. The researchers focused on three primary linguistic relations: Hyponymy (is-a), Antonymy (opposites), and Morphosemantic Links (e.g., verb-noun relationships).

The benchmark consists of 16,972 problems, categorized by WordNet relation and Question Pattern (QP), as shown in the table below.

Distribution of WordNet relations and Question Patterns in the benchmark.

A random sample of 169 problems (approximately 1% of the benchmark) was selected for detailed manual auditing. Each problem was evaluated for mapping correctness, proof validity, and the nature of the underlying knowledge used by the ATPs, such as Vampire and E.

Results and Analysis

The manual audit provided a granular view of the system's performance across different linguistic categories, revealing exactly where the logical reasoning process breaks down.

Detailed results of the manual audit of 169 reasoning problems across various categories.

Mapping Quality and the Adjective Gap

The analysis showed that 66% of the synsets were correctly mapped to SUMO. However, a significant "Adjective Gap" was identified. Adjectives are frequently mismapped to SUMO "Processes" rather than "Attributes," leading to a high failure rate in antonym-based reasoning. For instance, 65% of antonym problems featured incorrect mappings.

Hierarchy and Granularity

A major source of logical incompatibility arises when WordNet's hierarchy clashes with SUMO's constraints. For example, WordNet might classify a concept in a way that SUMO's logical axioms treat as mutually exclusive. Furthermore, even when mappings are correct, they are often "subsumptions"—where a specific WordNet term is mapped to a much broader SUMO concept. Only 14% of the analyzed problems featured precise, equivalent mappings.

Reasoning Bottlenecks

While 49% of the problems were successfully solved, the remaining failures were largely attributed to a lack of "middle-level" knowledge in SUMO. While SUMO excels at high-level abstractions (like "Entity"), it often lacks the specific details required to distinguish between closely related WordNet terms. In a minority of cases, theorem provers failed due to computational resource limits despite having the necessary knowledge available.

Conclusion

The study concludes that the primary bottleneck in commonsense reasoning is the "granularity gap" between the vastness of natural language and the relatively compact core of formal ontologies. The authors suggest that future work should focus on automated ontology augmentation, leveraging the rich semantic structure already present in WordNet to programmatically expand the logical depth and specificity of ontologies like SUMO.

Summary updated

BibTeX
@inproceedings{alvez2019commonsense,
  author = {Javier Álvez and Itziar Gonzalez-Dios and German Rigau},
  title = {{Commonsense Reasoning Using WordNet and SUMO: a Detailed Analysis}},
  booktitle = {GWC},
  year = {2019},
}
Figure