Abstract
In this paper, we consider the problem of machine reading task when the questions are in the form of keywords, rather than natural language. In recent years, researchers have achieved significant success on machine reading comprehension tasks, such as SQuAD and TriviaQA. These datasets provide a natural language question sentence and a pre-selected passage, and the goal is to answer the question according to the passage. However, in the situation of interacting with machines by means of text, people are more likely to raise a query in form of several keywords rather than a complete sentence. The keyword-based query comprehension is a new challenge, because small variations to a question may completely change its semantical information, thus yield different answers. In this paper, we propose a novel neural network system that consists a Demand Optimization Model based on a passage-attention neural machine translation and a Reader Model that can find the answer given the optimized question. The Demand Optimization Model optimizes the original query and output multiple reconstructed questions, then the Reader Model takes the new questions as input and locate the answers from the passage. To make predictions robust, an evaluation mechanism will score the reconstructed questions so the final answer strike a good balance between the quality of both the Demand Optimization Model and the Reader Model. Experimental results on several datasets show that our framework significantly improves multiple strong baselines on this challenging task.
Introduction
Machine Reading Comprehension (MRC) has seen significant advancements with datasets like SQuAD and TriviaQA. However, these benchmarks typically assume well-formed, natural language questions. In practical scenarios, such as search engine interactions or chatbot queries, users often provide keyword-based inputs (e.g., "obama presidency deeds") rather than complete sentences. This discrepancy introduces two major hurdles for AI models:
- Ambiguity: Small variations in keywords can drastically change the required answer.
- Incompleteness: Users often omit "inconsequential" words (like "do" or "during") that provide vital syntactic clues for models to locate an answer within a text.
The DORM Framework
To address these challenges, we propose the Demand-Optimized Reader Model (DORM). This framework consists of three primary components: a Demand Optimization Model (DOM), a Reader Model, and an Evaluation Mechanism.
- Demand Optimization Model (DOM): Utilizing a passage-aware neural machine translation engine, the DOM "translates" keyword queries into multiple full natural language questions. By attending to the provided passage, the model can infer missing context. A copying mechanism ensures that critical entities from the original query are preserved during reconstruction.
- Reader Model: This component takes the reconstructed questions and the passage as input. It employs a memory-network-based architecture with word and character-level embeddings, supplemented by linguistic features like Part-of-Speech (POS) and Named Entity Recognition (NER) tags, to identify the answer span.
- Evaluation Mechanism: Because the DOM generates multiple candidate questions via beam search, this module scores each one to weight the final answer prediction, ensuring robustness against lower-quality reconstructions.
Experimental Setup and Results
We evaluated DORM on keyword-pruned versions of SQuAD and TriviaQA. To simulate realistic keyword queries, stopwords were removed and TF-IDF pruning was applied to reduce questions to an average of 8 words or fewer.
DORM consistently outperformed strong baselines like BiDAF, DrQA, and MEMEN. On the modified SQuAD dataset, DORM achieved an F1 score of 72.62, significantly higher than the 65.04 achieved by a standard BiDAF model. Ablation studies confirmed that the Evaluation Mechanism is crucial, contributing a 3-5% performance gain.
Furthermore, we investigated the impact of the number of candidate questions generated by the DOM. We observed that the system's performance peaks when generating approximately 5 candidate questions. Beyond this point, the inclusion of lower-quality candidates introduces noise that can degrade accuracy.
Conclusion and Future Work
DORM effectively bridges the gap between keyword-based user intent and the requirements of standard MRC models by leveraging the context of the passage during query reconstruction. While the model shows strong performance, it faces difficulties with "Why" questions, which require more complex explanatory answers than "Who" or "When" queries. Future work will explore joint training of the generation and reading components to further improve the quality of the keyword-to-question mapping.
Summary updated