TL;DR
This paper introduces a novel semantic annotation scheme for question-answer pairs (QAPs) in natural conversations, aiming to capture both content and discourse function. They developed detailed guidelines and applied them to multilingual corpora, demonstrating its utility for both human annotation and initial machine learning experiments for automatic QAP classification.
What Problem Does This Paper Address?
Understanding the dynamics of human conversation is crucial for building more sophisticated dialogue systems, like chatbots and virtual assistants. A key component of this is analyzing how questions and answers function within a dialogue. While previous work has addressed aspects of question answering and dialogue act classification, there's a gap in a comprehensive scheme that captures the full range of QAPs found in natural, multilingual conversations. Existing approaches often lack the depth to characterize the semantic role of questioned constituents or the nuanced discourse contributions of answers, limiting their applicability for dialogue modeling.
Methods
The core contribution is a new semantic annotation scheme for QAPs. This scheme is built on two main "layers":
- Question/Answer Type: Categorizes the fundamental nature of the question (e.g., yes/no, wh-question, completion suggestion, phatic question) and the answer's relation to the question (e.g., positive, negative, feature answer, unrelated topic).
- Feature (Semantic Role): For certain question types (wh- and disjunctive), this layer annotates the semantic role of the questioned constituent (e.g., LOCATION, AGENT, TIME).
The authors developed detailed, explicit annotation guidelines to ensure consistency for human annotators. They also explored machine learning for automatic annotation, focusing on question type classification using a decision tree algorithm with hand-designed features.
Datasets
The authors annotated dialogues from three multilingual corpora:
- Saarbrucken Corpus of Spoken English (SCoSE): English, face-to-face conversations.
- CallFriend Corpus (Spanish): Spanish, phone conversations.
- Spoken Dutch Corpus (CGN): Dutch, phone conversations.
A subset of SCoSE was used to build a "gold standard" corpus for calculating inter-annotator agreement and for training machine learning models.
Key Insights and Ideas
- Multilingual Applicability: The scheme was designed and tested on English, Spanish, and Dutch, highlighting its potential for cross-lingual dialogue analysis.
- Semantic Role Annotation: Identifying the semantic role of the questioned constituent (e.g., "Where is the meeting?" -> LOCATION) provides a powerful way to globally characterize QAP content.
- Phatic Questions are Tricky: The authors found that distinguishing phatic questions (where social function is more important than semantic content) from other question types was a significant source of inter-annotator disagreement, suggesting this is a challenging area for both humans and machines.
- Feature Engineering for ML: Simple, hand-designed features like question length and the presence of specific words proved surprisingly effective for initial machine learning classification, outperforming a majority-class baseline.
Key Results
- Manual Annotation: 701 questions and 483 answers were annotated across the three languages. Inter-annotator agreement for question types was moderately high (Cohen's Kappa > 0.6), while agreement for answer types was lower.
- Machine Learning: A decision tree classifier achieved an accuracy of 0.73 and an F1-score of 0.58 for English question type classification, significantly outperforming a baseline.
Limitations and Future Work
- Inter-Annotator Agreement: While moderate, agreement on phatic questions and answer types was lower, indicating areas for refinement in guidelines or the scheme itself.
- Machine Learning Performance: The decision tree model's performance, while promising, is still limited, especially concerning phatic questions and utterances with interruptions. More advanced neural architectures (BOW, RNN) performed worse, suggesting a need for more data or better feature engineering.
- Future Work: The authors plan to expand the multilingual component, annotate more corpora, and adapt machine learning algorithms to improve automatic annotation, particularly for more complex or ambiguous cases. They also aim to incorporate discourse context into machine learning models.
Summary updated