TL;DR
This paper tackles the challenge of understanding follow-up questions in interactive question answering (IQA) systems, specifically focusing on those that continue the existing topic rather than shifting it. By analyzing real-world IQA dialogues, they develop machine learning models that leverage dialogue context and action-based features to predict correct answers, demonstrating improved accuracy over simpler approaches.
What Problem Does This Paper Address?
Interactive Question Answering (IQA) systems aim to provide answers through natural language dialogue, going beyond traditional QA systems that handle isolated questions. A key aspect of IQA is handling "Follow-Up Questions" (FU Qs) posed by users. These FU Qs can either be a continuation of the current topic or a shift to a new one. This paper focuses on the former: topic continuation FU Qs. The challenge lies in understanding the relationship between the FU Q and the preceding dialogue context (previous questions and answers) to accurately identify the correct answer from a set of candidates. Existing approaches often rely on artificial datasets or don't fully leverage the nuances of human-computer interaction in IQA.
Methods
The authors propose an empirical framework using Logistic Regression Models to tackle two main tasks:
- Modeling relations: Understanding how the FU Q's answer relates to the FU Q itself and the dialogue context.
- Answer prediction: Using this understanding to identify the correct answer among multiple candidates.
They extract features from dialogue snippets consisting of four turns: the previous user question (Q-1), the previous system answer (A-1), the follow-up question (FU Q), and the candidate answer (A0). Key feature sets include:
- TF/IDF Similarity: Lexical similarity scores between strings (e.g., FU Q and A0, or A-1 and A0) using TF/IDF weighting. This is a standard technique for measuring text similarity.
- Action Sequences: Features based on "library actions" identified in the questions and answers. They hypothesize that modeling the continuity or transitions between these actions can provide valuable context. This involves:
actionContinuityQA/actionContinuityAA: Binary features indicating if the same library action is present in the FU Q and A0, or A-1 and A0, respectively.ImProbQA/ImProbAA: Probabilities from smoothed 2-gram statistical language models trained on action sequences, capturing how likely certain action transitions are. This is a novel application of language modeling to action sequences in IQA.
Datasets
The study uses real IQA dialogues collected from a deployed system called BoB (a library help desk domain).
- The corpus consists of 948 user questions from 262 dialogue sessions.
- User FU Qs were manually annotated as either "topic continuation" (248 questions) or "topic shift" (150 questions).
- Answers from the system's repository (529 in total) were annotated with "library actions" (a flat list of 25 actions).
- For evaluation, 76 topic continuation FU Qs were selected after filtering out those following incorrect system answers.
This dataset is highlighted as being less constrained and potentially more difficult than synthesized datasets (like TREC context track data), offering insights into actual IQA dialogues.
Key Insights and Ideas
- Dialogue context matters, especially the system's last response: While previous work (like Yang et al., 2006) emphasized the importance of the previous user question (Q-1), this study found that the previous system answer (A-1) and its lexical similarity to the candidate answer (A0) were more predictive.
- Action-based features are informative: Modeling the underlying "actions" within questions and answers, and how they transition, provides significant information for identifying correct answers to topic continuation FU Qs.
- Combining features yields the best results: A "complete" model incorporating TF/IDF similarity and action-based features (specifically
tfIdfSimilarityQA,tfIdfSimilarityAA, andImProbQA) significantly outperformed models using only subsets of these features. - Not all actions are equally useful: While action-based features were generally beneficial,
actionContinuityAAwas found to be superfluous in most experiments, suggesting that the transition probability (ImProbQA) might be more critical than simple continuity.
Key Results
The paper evaluates models based on the mean rank of the correct answer among all 529 candidates. Lower ranks indicate better performance.
- The "complete" model (tfIdfSimilarityQA, tfIdfSimilarityAA, ImProbQA) achieved the lowest mean rank of 66.3 across five cross-validation splits.
- This "complete" model significantly outperformed reduced models:
- Reduced model 1 (no dialogue context, only FU Q and A0 features): Mean rank 80.2.
- Reduced model 2 (no action sequences, only TF/IDF): Mean rank 76.7.
- Reduced model 3 (baseline, only tfIdfSimilarityQA): Mean rank 98.8.
- In about half of the cases, the correct answer was ranked within the top 20 candidates by the "complete" model.
- The coefficients in the logistic regression model for the "complete" model indicated that
tfIdfSimilarityQAhad the largest positive contribution to the score, followed bytfIdfSimilarityAAand thenImProbQA.
Limitations and Future Work
- Limitations:
- The study acknowledges that the "difficult" cases (lowest-ranking third) often involve context-dependent or slightly varied FU Qs, suggesting a need for finer-grained classification of topic continuation.
- The current approach only marks one answer as "correct" per FU Q, which might be a simplification.
- The simple action detection algorithm for questions (stemming verb strings) can fail for elided verbs or unknown synonyms.
- Future Work:
- Implement a state-of-the-art topic-shift detection algorithm.
- Extend the system to distinguish three classes: topic shifts, fully specified topic continuations, and context-dependent topic continuations.
- Build dedicated logistic regression models for each sub-class of FU Qs, potentially using specific sets of predictors to improve overall ranking.
- Further explore the idea of action-action sequences, especially for context-dependent FU Qs.
Summary updated