Keyword-based Query Comprehending via Multiple Optimized-Demand Augmentation

Boyuan Pan, Hao Li, Zhou Zhao, Deng Cai, Xiaofei HearXiv 2017 · arXiv · OpenReview

Abstract

In this paper, we consider the problem of machine reading task when the questions are in the form of keywords, rather than natural language. In recent years, researchers have achieved significant success on machine reading comprehension tasks, such as SQuAD and TriviaQA. These datasets provide a natural language question sentence and a pre-selected passage, and the goal is to answer the question according to the passage. However, in the situation of interacting with machines by means of text, people are more likely to raise a query in form of several keywords rather than a complete sentence. The keyword-based query comprehension is a new challenge, because small variations to a question may completely change its semantical information, thus yield different answers. In this paper, we propose a novel neural network system that consists a Demand Optimization Model based on a passage-attention neural machine translation and a Reader Model that can find the answer given the optimized question. The Demand Optimization Model optimizes the original query and output multiple reconstructed questions, then the Reader Model takes the new questions as input and locate the answers from the passage. To make predictions robust, an evaluation mechanism will score the reconstructed questions so the final answer strike a good balance between the quality of both the Demand Optimization Model and the Reader Model. Experimental results on several datasets show that our framework significantly improves multiple strong baselines on this challenging task.

Introduction

Machine Reading Comprehension (MRC) has seen significant advancements with datasets like SQuAD and TriviaQA. However, these benchmarks typically assume well-formed, natural language questions. In practical scenarios, such as search engine interactions or chatbot queries, users often provide keyword-based inputs (e.g., "obama presidency deeds") rather than complete sentences. This discrepancy introduces two major hurdles for AI models:

  1. Ambiguity: Small variations in keywords can drastically change the required answer.
  2. Incompleteness: Users often omit "inconsequential" words (like "do" or "during") that provide vital syntactic clues for models to locate an answer within a text.

The DORM Framework

To address these challenges, we propose the Demand-Optimized Reader Model (DORM). This framework consists of three primary components: a Demand Optimization Model (DOM), a Reader Model, and an Evaluation Mechanism.

  1. Demand Optimization Model (DOM): Utilizing a passage-aware neural machine translation engine, the DOM "translates" keyword queries into multiple full natural language questions. By attending to the provided passage, the model can infer missing context. A copying mechanism ensures that critical entities from the original query are preserved during reconstruction.
  2. Reader Model: This component takes the reconstructed questions and the passage as input. It employs a memory-network-based architecture with word and character-level embeddings, supplemented by linguistic features like Part-of-Speech (POS) and Named Entity Recognition (NER) tags, to identify the answer span.
  3. Evaluation Mechanism: Because the DOM generates multiple candidate questions via beam search, this module scores each one to weight the final answer prediction, ensuring robustness against lower-quality reconstructions.
The DORM framework overview, showing the flow from keyword query to final answer via multiple reconstructed questions.

Experimental Setup and Results

We evaluated DORM on keyword-pruned versions of SQuAD and TriviaQA. To simulate realistic keyword queries, stopwords were removed and TF-IDF pruning was applied to reduce questions to an average of 8 words or fewer.

DORM consistently outperformed strong baselines like BiDAF, DrQA, and MEMEN. On the modified SQuAD dataset, DORM achieved an F1 score of 72.62, significantly higher than the 65.04 achieved by a standard BiDAF model. Ablation studies confirmed that the Evaluation Mechanism is crucial, contributing a 3-5% performance gain.

Furthermore, we investigated the impact of the number of candidate questions generated by the DOM. We observed that the system's performance peaks when generating approximately 5 candidate questions. Beyond this point, the inclusion of lower-quality candidates introduces noise that can degrade accuracy.

Performance (EM and F1) on SQuAD relative to the number of candidate questions generated.

Conclusion and Future Work

DORM effectively bridges the gap between keyword-based user intent and the requirements of standard MRC models by leveraging the context of the passage during query reconstruction. While the model shows strong performance, it faces difficulties with "Why" questions, which require more complex explanatory answers than "Who" or "When" queries. Future work will explore joint training of the generation and reading components to further improve the quality of the keyword-to-question mapping.

Summary updated

Abstract

In this paper, we consider the problem of machine reading task when the questions are in the form of keywords, rather than natural language. In recent years, researchers have achieved significant success on machine reading comprehension tasks, such as SQuAD and TriviaQA. These datasets provide a natural language question sentence and a pre-selected passage, and the goal is to answer the question according to the passage. However, in the situation of interacting with machines by means of text, people are more likely to raise a query in form of several keywords rather than a complete sentence. The keyword-based query comprehension is a new challenge, because small variations to a question may completely change its semantical information, thus yield different answers. In this paper, we propose a novel neural network system that consists a Demand Optimization Model based on a passage-attention neural machine translation and a Reader Model that can find the answer given the optimized question. The Demand Optimization Model optimizes the original query and output multiple reconstructed questions, then the Reader Model takes the new questions as input and locate the answers from the passage. To make predictions robust, an evaluation mechanism will score the reconstructed questions so the final answer strike a good balance between the quality of both the Demand Optimization Model and the Reader Model. Experimental results on several datasets show that our framework significantly improves multiple strong baselines on this challenging task.
BibTeX
@misc{pan2017keyword,
  author = {Boyuan Pan and Hao Li and Zhou Zhao and Deng Cai and Xiaofei He},
  title = {{Keyword-based Query Comprehending via Multiple Optimized-Demand Augmentation}},
  year = {2017},
  eprint = {1711.00179},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.00179},
}
Figure