A Generate-Validate Approach to Answering Questions about Qualitative Relationships

Arindam Mitra, Chitta Baral, Aurgho Bhattacharjee, Ishan ShrivastavaarXiv 2019 · OpenReview

Abstract

Qualitative relationships describe how increasing or decreasing one property (e.g., altitude) affects another (e.g., temperature). They are an important aspect of natural language question answering and are crucial for building chatbots or voice agents. Traditionally, semantic parsers have been used to convert natural language input into logical forms. However, these parsers often struggle with "cold start" problems when target languages change. In this paper, we present a generate-validate framework (gvQPS) that generates natural language descriptions of logical forms and validates them against input text using Textual Entailment models. Our method leverages transfer learning from general-purpose datasets and outperforms the state-of-the-art on the QUAREL benchmark by a margin of 7.93%.

Introduction

Answering questions about qualitative relationships—how a change in one property affects another—is a significant challenge for AI. Traditionally, this is solved using semantic parsing, where a model translates a natural language question directly into a domain-specific logical form. However, semantic parsing has a major limitation: every time the application or the target logical language changes, the parser must learn new symbols and syntax from scratch. It cannot easily leverage general linguistic knowledge because the output is not human language.

This research addresses the brittleness of these parsers by shifting the burden from generating complex logic to validating natural language hypotheses. By converting potential logical answers back into natural language, we can utilize pre-trained models that already "understand" English.

The gvQPS Framework

The proposed gvQPS (generate-validate Qualitative Problem Solver) algorithm follows a three-step process:

1. Generate

Instead of writing correct logic directly, the system uses predefined templates to create a massive set of natural language hypotheses. For a problem involving friction and heat, it might generate sentences like "carpet has more friction" or "skin has less friction" based on noun phrases found in the text. This step bridges the gap between abstract logic and human language.

Templates used to generate natural language hypotheses from qualitative properties.

2. Validate

The system uses Natural Language Inference (NLI) models, specifically BERT and ESIM, to score these hypotheses. It evaluates the likelihood of a hypothesis (e.g., "carpet has more friction") being true given the provided story. This allows the model to use transfer learning from massive general-purpose datasets like the Stanford Natural Language Inference (SNLI) corpus.

3. Answer Generation

The system compares the scores of hypotheses related to different multiple-choice options. It utilizes a Qualitative Relations Knowledge Base (QRKB) to bridge the gap between the "given facts" in the story and the "claims" made in the answer choices. The QRKB contains 25 qualitative relations, such as "friction is proportional to heat."

Experimental Results

The gvQPS approach was evaluated on the QUAREL benchmark, which contains 2,771 multiple-choice story questions. The results demonstrate that framing the problem as an entailment task significantly improves performance over traditional semantic parsing.

Performance comparison of different model configurations on the QUAREL dataset.

As shown in the table above, the best-performing model configuration achieved an accuracy of 76.63% on the test set. This is a substantial improvement over the previous state-of-the-art, QUASP+ (68.7%), and the standard semantic parser, QUASP (56.1%). The use of BERT combined with SNLI pre-training provided the most significant performance gains, confirming that general linguistic pre-training is highly effective for specific reasoning tasks when framed correctly.

Conclusion and Future Work

The generate-validate framework successfully overcomes the "target language" gap inherent in traditional semantic parsing. By translating logic into natural language templates, the authors turned a translation problem into an entailment problem, enabling the use of powerful pre-trained models.

Error analysis suggests that the current bottleneck is "given fact" understanding—correctly extracting the underlying qualitative rules from the initial premise. Future work will explore applying this framework to other domains that rely on semantic parsing, such as math word problems or Text-to-SQL tasks, where the target language barrier remains a significant hurdle.

Summary updated

Abstract

Qualitative relationships describe how increasing or decreasing one property (e.g. altitude) affects another (e.g. temperature). They are an important aspect of natural language question answering and are crucial for building chatbots or voice agents where one may enquire about qualitative relationships. Recently a dataset about question answering involving qualitative relationships has been proposed, and a few approaches to answer such questions have been explored, in the heart of which lies a semantic parser that converts the natural language input to a suitable logical form. A problem with existing semantic parsers is that they try to directly convert the input sentences to a logical form. Since the output language varies with each application, it forces the semantic parser to learn almost everything from scratch. In this paper, we show that instead of using a semantic parser to produce the logical form, if we apply the generate-validate framework i.e. generate a natural language description of the logical form and validate if the natural language description is followed from the input text, we get a better scope for transfer learning and our method outperforms the state-of-the-art by a large margin of 7.93%.
BibTeX
@misc{mitra2019generate,
  author = {Arindam Mitra and Chitta Baral and Aurgho Bhattacharjee and Ishan Shrivastava},
  title = {{A Generate-Validate Approach to Answering Questions about Qualitative Relationships}},
  year = {2019},
  eprint = {1908.03645},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.03645},
}
Figure