The Learning-Knowledge-Reasoning Paradigm for Natural Language Understanding and Question Answering

Arindam MitraICLP (Technical Communications) 2018 · DOI

TL;DR

This paper introduces a novel "Learning-Knowledge-Reasoning" (LKR) paradigm for question answering (QA) that tackles questions requiring more than just direct text retrieval. By combining a semantic parser, inductive logic programming (ILP) for knowledge acquisition, and an Answer Set Programming (ASP) reasoner, the system learns commonsense knowledge from data to answer complex questions. This approach aims to bridge the gap between statistical NLP and symbolic reasoning for more robust AI agents.

What Problem Does This Paper Address?

Developing AI agents that can truly understand text and answer questions that go beyond simple fact extraction is a long-standing goal in Artificial Intelligence. Many existing QA challenges, like those in science, math word problems, and general text comprehension (e.g., bAbI), require not only understanding the provided text but also leveraging external knowledge, often commonsense knowledge, and performing sophisticated reasoning. Manually encoding this vast amount of commonsense knowledge is impractical. This paper addresses the challenge of automatically acquiring this missing knowledge from question-answering datasets.

Methods

The core of the proposed solution is the Learning-Knowledge-Reasoning (LKR) paradigm, which consists of three main components:

  1. Semantic Parser (T): This component converts natural language text into a formal logical representation. The paper mentions using an AMR (Abstract Meaning Representation) parser from a statistical NLP layer, followed by a rule-based procedure to convert AMR graphs into the syntax of Event Calculus.
  2. Inductive Logic Programming (ILP) Module (L): This is the knowledge acquisition engine. The paper adapts a standard ILP task to a "mutually distinct examples" framework (ILPDE). Given background knowledge (B), a language bias (M), and a training dataset (D) of examples (each with observations, positive, and negative ground literals), the ILP module learns a hypothesis (H) – essentially, missing knowledge in the form of ASP rules. This learned knowledge H, along with the background knowledge B, is then used for reasoning.
  3. Reasoning Engine (R): The paper uses Answer Set Programming (ASP) for knowledge representation and reasoning. ASP provides a declarative way to express knowledge and perform logical inference. The learned hypothesis H and background knowledge B are expressed as ASP rules, and the ASP solver computes the answer to a given query.

The paper also highlights the use of Event Calculus as a temporal logic for representing events and their effects, and its domain-independent axioms (like the law of inertia) are part of the background knowledge.

Datasets

The paper primarily uses examples from the bAbI question answering challenge for demonstrating and evaluating their approach. They mention that the training datasets for bAbI tasks typically contain 1000 examples. They also refer to the Aristo science challenge and Project Euclid's math word problems as examples of tasks requiring external knowledge.

Key Insights and Ideas

  • Learning Commonsense Knowledge is Crucial: The paper strongly emphasizes that for many QA tasks, the answer isn't directly in the text but requires commonsense knowledge (e.g., the effect of actions, part-whole relations).
  • ILP as a Knowledge Acquisition Tool: Inductive Logic Programming is presented as a powerful method to automatically learn this missing commonsense knowledge from data, rather than relying on manual encoding.
  • Bridging Symbolic and Statistical AI: The LKR paradigm elegantly combines statistical NLP (for parsing) with symbolic reasoning (ASP) and machine learning (ILP) to create a more comprehensive QA system.
  • Adaptation of ILP for Machine Learning Datasets: The introduction of the "mutually distinct examples" framework for ILP is a key methodological contribution, making ILP more suitable for the context-dependent nature of typical ML datasets.
  • ASP for Declarative Reasoning: Using ASP provides a robust and declarative framework for representing both background and learned knowledge, simplifying the reasoning process.

Key Results

The paper states that their system, using the LKR method, outperforms all deep learning systems for the bAbI challenge on task 8. While specific quantitative results (e.g., accuracy percentages) are not detailed in the provided text, this comparative performance is highlighted as a significant achievement. They also mention that the learned rules (shown in Table 4 for task 8) are "human readable," which is a desirable qualitative outcome.

Limitations and Future Work

  • Current Focus: The authors are currently applying this framework to word arithmetic problem solving, aiming to learn human-readable knowledge that guides the selection and ordering of arithmetic formulas.
  • Scalability: While the paper mentions an iterative and incremental algorithm for ILPDE, the scalability to very large datasets is an implicit area for future exploration.
  • Generalizability: The success on bAbI task 8 is promising, but further work is needed to demonstrate the generalizability of the LKR paradigm across a wider range of QA tasks and knowledge domains.
  • Learning Logic Programs: The conclusion poses a fundamental question about whether machines learning logic programs can achieve better accuracy than existing statistical methods, suggesting a broader research agenda focused on learning from large datasets.

Summary updated

BibTeX
@inproceedings{mitra2018learning,
  author = {Arindam Mitra},
  title = {{The Learning-Knowledge-Reasoning Paradigm for Natural Language Understanding and Question Answering}},
  booktitle = {ICLP (Technical Communications)},
  year = {2018},
  doi = {10.4230/oasics.iclp.2018.19},
}
Figure