TL;DR
This paper presents a live demonstration comparing a neural network-based QA system (LSTM) against a traditional information retrieval baseline (BM25) on a massive community QA dataset. The goal is to empirically assess whether state-of-the-art neural models truly outperform traditional methods in terms of both answer quality and retrieval speed when scaled to real-world data sizes.
What Problem Does This Paper Address?
The field of open-domain question answering (QA) has seen significant advancements with the rise of neural network models, particularly Long Short-Term Memory (LSTM) networks, fueled by the availability of large community question answering (cQA) datasets. It's often claimed that these neural systems significantly outperform traditional information retrieval (IR) baselines like BM25. However, many studies supporting this claim rely on relatively small, curated datasets, such as those from older TREC QA tracks. This raises a crucial question: how do these advanced neural QA systems perform when faced with the sheer scale and diversity of real-world cQA data, both in terms of answer quality and response time? This paper aims to bridge this gap by providing a direct, hands-on comparison.
Methods
The authors compare two distinct QA systems:
- LSTM-based QA: This system utilizes a bidirectional LSTM (BLSTM) architecture. For each question-answer pair in the training data, both the question and answer are converted into fixed-size vector representations. The BLSTM processes word embeddings (pre-trained using skip-gram on Japanese Wikipedia) to generate these representations. A triplet loss function is used during training to minimize the distance between a question and its correct answer while maximizing the distance to incorrect answers. For retrieval, answers are indexed using a spatial indexing tool (NGT) after being converted into vector representations by the BLSTM.
- BM25: This is a well-established IR baseline. The authors use Elasticsearch with default settings. For retrieval, queries are formulated by extracting nouns, verbs, and adjectives from the input question after morphological analysis. All answers in the cQA knowledge source are indexed for retrieval.
The core demonstration system is bilingual, accepting English questions (which are machine-translated to Japanese) and presenting Japanese answers (which are then translated back to English).
Datasets
- Training Data: The LSTM model is trained on a large subset of Yahoo! Chiebukuro (Japanese Yahoo! Answers), comprising over 11 million questions and over 27.4 million answers.
- Q&A Knowledge Source: Both the LSTM and BM25 systems retrieve answers from a dataset of 100,000 questions and 241,994 answers, also derived from Yahoo! Chiebukuro. The authors mention plans to expand this dataset for the actual demonstration.
Key Insights and Ideas
- The "Real-World Scale" Question: The paper directly tackles the critical question of whether the claimed superiority of neural QA systems over traditional IR methods holds true when scaled to millions of data points, a scenario often absent in prior research.
- Hands-on Comparison for SIGIR Attendees: The demonstration format is a key idea, allowing researchers to directly interact with and compare the systems, fostering a deeper understanding beyond abstract metrics.
- Bilingual Demonstration: Leveraging machine translation to handle English input and output for a primarily Japanese system is a practical approach to making the demonstration accessible to a global audience.
Key Results
- Efficiency: The LSTM component demonstrates impressive retrieval speeds. Retrieving the top 10 answers takes an average of 5.76 milliseconds, and retrieving the top 1000 answers takes 41.31 milliseconds from a candidate set of 155,648. This suggests that, with proper indexing, neural models can be efficient.
- Effectiveness (Qualitative): Figure 3 provides a qualitative comparison of answers for a sample question. The LSTM system provides answers that are more direct and contextually relevant to the nuances of the question (e.g., addressing the legality of downloading music and anime). The BM25 system, while retrieving relevant information, sometimes provides answers that are less directly responsive or focus on tangential aspects.
Limitations and Future Work
- Data Size for Demonstration: While the training data is massive, the Q&A knowledge source used for the actual demonstration at the time of writing was 100,000 questions, with plans to expand it. The authors acknowledge that the conclusion about effectiveness might depend on the scale and diversity of this retrieval corpus.
- Answer Length Constraint: Due to memory limitations, the LSTM system only indexed answers shorter than 50 tokens. This means longer, potentially more comprehensive answers might not be retrievable by the LSTM component.
- Future Work: The paper implicitly suggests that further investigation into how neural models generalize to larger, more diverse datasets is warranted. The authors also pose a broader question for the community: "Given the abundance of data, are neural network-based QA systems going to completely replace traditional IR-based systems in the near future?" This invites ongoing research and discussion.
Summary updated