Knowledge-driven Natural Language Understanding of English Text and its Applications

Kinjal Basu, Sarat Chandra Varanasi, Farhad Shakerin, Joaquín Arias, Gopal GuptaAAAI 2021 · DOI

Abstract

This paper introduces a novel framework for Natural Language Understanding (NLU) that shifts the focus from statistical pattern matching to logic-based "true understanding." By mapping English text to knowledge primitives using VerbNet and reasoning over them with Answer Set Programming (ASP), the authors demonstrate a system that achieves near-perfect accuracy on complex reasoning tasks while providing human-readable justifications.

Introduction

Most modern NLU systems, including Large Language Models (LLMs) and deep learning-based QA models, operate as "black boxes." While they excel at identifying patterns in massive datasets, they often lack a grounded "world model." This deficiency leads to two major issues: a lack of explainability and failures in non-monotonic reasoning—the ability to revise conclusions when new information arrives (e.g., updating a person's location from the kitchen to the bedroom as a story progresses).

The authors address these gaps by treating natural language more like a programming language with formal denotational semantics. In this framework, every sentence maps to a specific, logical meaning that can be used for rigorous reasoning.

Methodology: From Text to Logic

The core of the proposed system is a pipeline that converts raw text into a "semantic algebra" of knowledge.

The pipeline for converting natural language sentences into ASP-based semantic representations using VerbNet.

The process involves several key components:

  • Semantic Mapping via VerbNet: The system utilizes VerbNet, a comprehensive hierarchical lexicon of English verbs. After parsing a sentence (e.g., using Stanford CoreNLP), the system matches the syntax tree against VerbNet’s "frames." For instance, the verb "grab" is automatically mapped to predicates like contact, cause, and transfer.
  • Answer Set Programming (ASP): The extracted meaning is represented as ASP code, a declarative logic language designed for complex search and reasoning problems.
  • s(CASP) Engine: Reasoning is performed by s(CASP), a goal-directed execution engine for ASP. It supports "negation-as-failure," allowing the system to handle commonsense defaults and revise beliefs as new facts emerge.

The authors implemented this framework in two distinct systems: SQuARE for question answering and StaCACK, a stateful chatbot that maintains conversation context via a Finite State Machine.

Evaluation and Results

The framework was evaluated using the Facebook bAbI datasets, which consist of tasks designed to test specific reasoning capabilities like counting, induction, and pathfinding.

  • SQuARE: Achieved 100% accuracy on 18 out of 20 bAbI tasks, with an overall accuracy of 99.9%. It significantly outperformed Memory Neural Networks (MemNN), particularly in tasks involving positional reasoning and pathfinding.
  • StaCACK: Achieved 100% accuracy on bAbI dialog tasks, including "Out-of-Vocabulary" (OOV) scenarios where traditional machine learning models often fail because they haven't seen specific words during training.

One of the most significant advantages of this approach is its inherent explainability. For every answer, the system generates a natural language justification based on its internal proof tree, showing exactly which events and logical steps led to the conclusion.

An example of a natural language justification generated from the system's internal logic-based proof tree.

Discussion and Key Insights

The system's success stems from several innovative ideas:

  • Denotational Semantics for English: Treating English sentences as code creates a domain-independent way to represent knowledge.
  • Non-Monotonic Reasoning: The system handles the "law of inertia" logically. If an object is possessed at time $T_1$, the system assumes possession continues until a specific "transfer" event is recorded.
  • Zero-Shot Reasoning: Because the system relies on a general-purpose lexicon and fixed commonsense rules, it does not require re-training for different tasks; it only needs the relevant axioms.

Limitations and Future Work

Despite its high performance, the system currently faces some constraints. It relies heavily on the accuracy of initial syntactic parsing; highly complex or ungrammatical sentences can disrupt the mapping. Additionally, while verb semantics are automated via VerbNet, some high-level commonsense rules still require manual encoding in ASP. Future work aims to scale the system from "toy" datasets to handle more complex, real-world stories and open-domain conversations.

Summary updated

BibTeX
@inproceedings{basu2021knowledge,
  author = {Kinjal Basu and Sarat Chandra Varanasi and Farhad Shakerin and Joaquín Arias and Gopal Gupta},
  title = {{Knowledge-driven Natural Language Understanding of English Text and its Applications}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i14.17488},
}
Figure