Abstract
Indirect answers are replies to polar questions without the direct use of word cues such as 'yes' and 'no'. Humans are very good at understanding indirect answers, such as 'I gotta go home sometime', when asked 'You wanna crash on the couch?'. Understanding indirect answers is a challenging problem for dialogue systems. In this paper, we introduce a new English corpus to study the problem of understanding indirect answers. Instead of crowdsourcing both polar questions and answers, we collect questions and indirect answers from transcripts of a prominent TV series and manually annotate them for answer type. The resulting dataset contains 5,930 question-answer pairs. We release both aggregated and raw human annotations. We present a set of experiments in which we evaluate Convolutional Neural Networks (CNNs) for this task, including a cross-dataset evaluation and experiments with learning from disagreements in annotation. Our results show that the task of interpreting indirect answers remains challenging, yet we obtain encouraging improvements when explicitly modeling human disagreement.
Introduction
In natural conversation, humans rarely stick to simple "yes" or "no" responses. When asked, "Do you want to go to the gym?" a friend might reply, "I’m still in my pajamas." While a human instinctively interprets this as a "No," this pragmatic leap remains a significant hurdle for dialogue systems. The meaning is not found in the literal words but in the context and intent.
Existing datasets often rely on artificial crowdsourced prompts. Furthermore, standard NLP approaches typically treat annotator disagreement as noise to be filtered out. This research introduces FRIENDS-QIA, a dataset derived from the TV show Friends, and argues that in the subtle world of indirect speech, disagreement is a valuable signal reflecting the inherent ambiguity of language.
Methodology
The researchers developed a classification system to categorize indirect answers into four labels: Yes, No, Yes subject to conditions, and Neither.
Base Architecture
The core model utilizes Convolutional Neural Networks (CNNs). The team evaluated the performance of both traditional word embeddings (GloVe) and contextualized embeddings (BERT).
Modeling Disagreement: The Crowd Layer
The most significant technical innovation is the implementation of Deep Learning from Crowds (DLFC). Rather than training solely on a "gold" label derived from majority voting, the model incorporates a specialized "Crowd Layer." This layer learns the specific biases and tendencies of individual annotators, allowing the system to capture the uncertainty present when humans themselves disagree on a character's intent.
Datasets
The study utilizes two primary datasets:
- FRIENDS-QIA (New): Comprising 5,930 question-answer pairs extracted from all 10 seasons (228 episodes) of Friends. It features linguistically rich dialogue, including sarcasm, irony, and mid-sentence shifts, and includes raw annotation data.
- CIRCA: A larger existing dataset (34,268 pairs) used for cross-dataset comparison.
To prevent the models from relying on simple lexical cues, the authors manually removed direct markers like "Yeah" or "Nope" from the answers, forcing the AI to interpret the underlying pragmatic meaning.
Key Results and Insights
- BERT Superiority: Transitioning from GloVe to BERT embeddings resulted in a substantial performance jump, increasing accuracy from approximately 55% to over 64% on the development set.
- The Value of Disagreement: Explicitly modeling human disagreement via the Crowd Layer improved the Macro F1-score from 49.16 to 55.00 for the BERT-based model. This was particularly effective for nuanced classes like "Yes, subject to conditions."
- Answer-Only Effectiveness: Surprisingly, models trained only on the answer often outperformed those trained only on the question, suggesting the answer carries the primary pragmatic weight in indirect communication.
- Natural vs. Crowdsourced Data: Models trained on the crowdsourced CIRCA dataset struggled with the Friends data, highlighting that scripted TV dialogue—with its character quirks and deep context—is significantly more complex than artificial prompts.
Conclusion and Future Work
While the results are encouraging, the task remains difficult. Current models are limited by a lack of broader context, typically looking at only a single question-answer pair. Future improvements likely require multi-modal approaches that incorporate prosody (tone of voice) and body language, as a sarcastic "Sure!" is pragmatically opposite to an enthusiastic one. Moving toward models that can "see" and "hear" context will be essential for truly understanding human dialogue.
Summary updated