TL;DR
This paper introduces CONVEX, a novel unsupervised method for conversational question answering (QA) over knowledge graphs (KGs). CONVEX addresses the challenge of incomplete and ungrammatical follow-up questions by intelligently expanding the KG context, outperforming existing baselines and question completion strategies. The authors also release CONVQUESTIONS, a new benchmark dataset for evaluating conversational KG-QA.
What Problem Does This Paper Address?
Traditional Question Answering (QA) systems, especially those operating over structured knowledge graphs (KGs), typically expect well-formed, self-contained questions. However, in real-world conversational settings, users often ask follow-up questions that are incomplete, relying on implicit context from previous turns. These questions might omit entities or predicates, use ungrammatical phrasing, or shift focus subtly. Existing KG-QA systems struggle with this ambiguity, as they cannot directly map such fragmented queries to KG facts. This paper tackles the problem of enabling KG-QA systems to handle these natural, conversational follow-up questions by effectively leveraging the conversation history.
Methods
CONVEX employs a two-stage approach: context expansion and answer ranking.
- Context Expansion: For the initial question, a standard KG-QA system is used. The entities from the question and its answer, along with their KG connections, form the initial context subgraph. For subsequent follow-up questions, CONVEX dynamically expands this context. Instead of blindly adding all neighbors of the current context nodes (which can lead to combinatorial explosion), CONVEX uses a "judicious" expansion strategy. It identifies "frontier nodes" – the most relevant nodes in the KG neighborhood – based on three signals:
- Relevance to Question: Similarity of node labels to question words (using word embeddings).
- Relevance to Context: Proximity to existing entities in the conversation history (weighted by turn recency).
-
KG Priors: Inherent salience of nodes in the KG (approximated by frequency). These signals are aggregated using Fagin's Threshold Algorithm to select the top-k frontier nodes. The KG facts connected to these frontiers are then added to expand the context graph.
-
Answer Ranking: Once the context graph is expanded, CONVEX ranks candidate answers within this expanded graph. The ranking considers the proximity of candidate answers to both the frontier nodes and the existing context entities, again using weighted scores. The top-ranked candidate is returned as the answer.
The method is unsupervised, meaning it doesn't require labeled training data for the conversational aspect.
Datasets
- CONVQUESTIONS: The authors introduce a new crowdsourced benchmark dataset called CONVQUESTIONS. It comprises approximately 11,200 conversations across five domains: "Books," "Movies," "Soccer," "Music," and "TV Series." These conversations were generated by crowdworkers on Amazon Mechanical Turk, with each conversation consisting of five turns (a question and its answer, followed by four follow-up questions and their answers). The dataset is designed to reflect natural user information-seeking behavior and poses significant challenges due to incomplete cues, comparisons, aggregations, and temporal reasoning. Answers are grounded in Wikidata.
Key Insights and Ideas
- "Look before you hop": The core idea is to intelligently select which parts of the KG to expand the context with, rather than blindly adding everything. This "judicious expansion" is crucial for managing complexity and maintaining focus.
- Context is King (and Queen): The success of conversational QA hinges on effectively leveraging past turns. CONVEX shows that incorporating entities and predicates from previous turns, weighted by recency, is vital.
- Frontiers are Essential: Identifying "frontier nodes" as key expansion points is more effective than directly searching for answers in a broad neighborhood. This two-stage process (frontier identification then answer ranking) helps overcome the ambiguity of follow-up questions.
- Unsupervised Advantage: CONVEX's unsupervised nature makes it highly adaptable to new domains and enterprise settings without requiring extensive retraining.
Key Results
- CONVEX significantly improves the performance of stand-alone KG-QA systems (like QAnswer and Platypus) on follow-up questions, boosting MRR scores from very low (e.g., 0.011-0.064) to much higher (e.g., 0.172-0.264) across domains.
- CONVEX outperforms naive conversational models like "star-join" (always referencing the initial entity) and "chain-join" (always referencing the previous answer), often by over 100%.
- The "frontier-less" approach (directly searching for answers without identifying frontiers) resulted in degraded performance, highlighting the importance of the frontier selection step.
- CONVEX maintains its performance over conversation turns, demonstrating robustness against potential errors in earlier turns.
- Pre-trained neural models (like D2A trained on CSQA) did not perform well out-of-the-box on CONVQUESTIONS, suggesting that existing benchmarks might not fully capture the complexity of real-world conversational QA.
Limitations and Future Work
- Error Analysis: The paper notes that the majority of errors (80-90%) occur during the context expansion phase, indicating room for improvement in frontier scoring. Answer ranking also has potential for improvement by better aggregating proximity signals.
- Future Work: The authors suggest exploring more informed frontier scoring strategies and better aggregation methods for answer ranking. They also highlight the potential for future work in combining KGs with text for even richer QA capabilities. The CONVQUESTIONS benchmark itself is presented as a resource to drive future research in this area.
Summary updated