Black-box Testing of First-Order Logic Ontologies Using WordNet

Javier Álvez, Paqui Lucio, German RigauarXiv 2017 · OpenReview

Abstract

Artificial Intelligence aims to provide computer programs with commonsense knowledge to reason about our world. This paper offers a new practical approach towards automated commonsense reasoning with first-order logic (FOL) ontologies. We propose a new black-box testing methodology of FOL SUMO-based ontologies by exploiting WordNet and its mapping into SUMO. Our proposal includes a method for the (semi-)automatic creation of a very large benchmark of competency questions and a procedure for its automated evaluation by using automated theorem provers (ATPs). Applying different quality criteria, our testing proposal enables a successful evaluation of a) the competency of several translations of SUMO into FOL and b) the performance of various automated ATPs. Finally, we also provide a fine-grained and complete analysis of the commonsense reasoning competency of current FOL SUMO-based ontologies.

Introduction

In the quest for Artificial Intelligence with "commonsense," researchers use ontologies—formal frameworks that define concepts and the relationships between them. The Suggested Upper Merged Ontology (SUMO) is one of the largest and most comprehensive, but utilizing it for automated reasoning requires translating its high-level axioms into First-Order Logic (FOL).

The primary bottleneck in ontology development is verification. Traditionally, researchers use Competency Questions (CQs)—logical conjectures the ontology should be able to prove—to determine if an ontology is "competent." However, manual creation of CQs is slow, expensive, and difficult to scale. Most benchmarks consist of only a few dozen manual questions, which is insufficient for a system as massive as SUMO. This research addresses the need for a massive, automated, and objective way to test if these digital "brains" actually understand the world.

Methodology

The authors propose a black-box testing approach, where test cases are derived from an external source of truth (WordNet) rather than the internal structure of the ontology.

The WordNet-SUMO Bridge

The methodology leverages the existing mapping between WordNet, a massive lexical database of English, and SUMO. By treating WordNet's semantic relations as a ground truth, the authors can generate formal logical conjectures.

Automated Conjecture Generation

Four main "question patterns" were developed based on WordNet’s internal structure:

  1. Multiple Mapping: Checking logical compatibility when a word maps to multiple SUMO concepts.
  2. Antonymy: Proving that an object cannot simultaneously possess properties defined as opposites in WordNet (e.g., "hot" and "cold").
  3. Event Patterns: Linking nouns and verbs that refer to the same process.
  4. Process Patterns: Using morphosemantic links to test roles like agent, instrument, and result.
Example of a formal axiom in SUMO defining a TransitiveRelation.

For every generated conjecture (a "truth-test"), the system also generates its negation (a "falsity-test") to ensure the ontology doesn't simply prove everything due to inconsistency.

Datasets and Experimental Setup

The study utilizes several versions of SUMO translated into FOL, comparing them against the newly generated benchmark.

  • WordNet v3.0: Source for lexical relations.
  • Adimen-SUMO (v2.2 and v2.6): Specific FOL translations maintained by the authors, with v2.6 containing over 100 axiom corrections.
  • TPTP-SUMO: A version from the "Thousands of Problems for Theorem Provers" library.
Comparison of the size and composition of different SUMO-based ontologies.

The authors produced a set of 7,505 problems, resulting in over 15,010 individual CQs, making it the largest set of competency questions ever proposed for SUMO. These were processed by state-of-the-art Automated Theorem Provers (ATPs) like Vampire and E.

Key Results

The testing framework successfully identified significant differences in competency across ontologies and provers:

  • Ontology Evolution: Adimen-SUMO v2.6 emerged as the most competent, passing significantly more tests than its predecessors. This demonstrates that targeted axiom fixes lead to measurable improvements in logical reasoning.
  • Prover Performance: Vampire v2.6 was the most effective prover overall, finding 2,327 proofs. Interestingly, newer versions of provers did not always outperform older ones on these specific commonsense tasks.
  • Mapping Validation: A manual audit revealed that while the system is robust, mapping adjectives remains a challenge, as many are incorrectly mapped to SUMO "processes" rather than "attributes."

Conclusion

This research demonstrates that informal lexical knowledge in WordNet can be systematically transformed into formal logical constraints for stress-testing ontologies. While the methodology successfully scaled testing from dozens to thousands of questions, limitations remain regarding adjective mapping and ontology incompleteness. Future work will explore other relations like meronymy and integrate additional knowledge sources like ConceptNet or FrameNet to further enrich the test suite.

Summary updated

Abstract

Artificial Intelligence aims to provide computer programs with commonsense knowledge to reason about our world. This paper offers a new practical approach towards automated commonsense reasoning with first-order logic (FOL) ontologies. We propose a new black-box testing methodology of FOL SUMO-based ontologies by exploiting WordNet and its mapping into SUMO. Our proposal includes a method for the (semi-)automatic creation of a very large benchmark of competency questions and a procedure for its automated evaluation by using automated theorem provers (ATPs). Applying different quality criteria, our testing proposal enables a successful evaluation of a) the competency of several translations of SUMO into FOL and b) the performance of various automated ATPs. Finally, we also provide a fine-grained and complete analysis of the commonsense reasoning competency of current FOL SUMO-based ontologies.
BibTeX
@misc{alvez2017black,
  author = {Javier Álvez and Paqui Lucio and German Rigau},
  title = {{Black-box Testing of First-Order Logic Ontologies Using WordNet}},
  year = {2017},
  eprint = {1705.10217},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1705.10217},
}
Figure