TL;DR
This paper tackles the challenge of automatically classifying "sluices" (bare wh-phrases like "Who?" or "When?") in dialogue. They developed a machine learning approach that achieves around 90% accuracy by learning from heuristic principles derived from a corpus study, demonstrating that these principles are well-motivated and can be learned automatically.
What Problem Does This Paper Address?
Dialogue often features elliptical constructions, where parts of a sentence are omitted but understood from context. One common type is "sluicing," where a bare wh-phrase stands in for a full interrogative clause (e.g., "Cassie: I know someone who's a good kisser. Catherine: Who?"). While theoretical linguists have studied sluicing, computational approaches, especially in the context of dialogue, have been less explored. This paper focuses on classifying these bare wh-phrases into different "dialogue types" (like direct questions, requests for clarification, or reprises) to better understand and process conversational turns. The authors highlight that direct and reprise sluices together constitute a significant portion of wh-queries in dialogue, making their accurate classification crucial for dialogue systems.
Methods
The authors employ a two-pronged approach: first, a corpus-based linguistic analysis to derive heuristic principles, and second, a machine learning framework to learn classification rules from these principles.
- Corpus Study and Heuristic Derivation: They analyzed a large corpus of dialogue transcripts (British National Corpus - BNC) to identify different types of sluices and their typical interpretations. Based on patterns observed in this corpus, they formulated "heuristic principles" for classifying sluices into categories like "Direct," "Reprise," and "Clarification." These principles were then formalized as probabilistic Horn clauses.
- Machine Learning: The predicates from these probabilistic Horn clauses were used to create features for annotating a sample of sluices. Two machine learning algorithms were then applied:
- SLIPPER: A rule-based learning algorithm that generates transparent, if-then rules, which the authors note closely resemble their derived Horn clauses.
- TiMBL: A memory-based learning algorithm that classifies instances by generalizing from similar examples in the training data.
Datasets
The primary dataset used is the British National Corpus (BNC), specifically its dialogue transcripts.
- The authors extracted approximately 10 million words of dialogue.
- They identified and collected all bare sluices, totaling 5343 instances.
- For annotation and machine learning, they created two samples:
- Sample 1: 365 sluices, including all instances of "how" and "which," plus 50 of each remaining class.
- Sample 2: 300 sluices, focusing on the remaining classes.
- For the machine learning experiments, they combined instances with three-way agreement among annotators and instances with agreement between the two most expert annotators, excluding "unclear" cases. This resulted in a dataset of 351 data points for training and testing.
Key Insights and Ideas
- Sluices are not monolithic: The study reveals significant correlations between the type of wh-word (what, why, who, etc.) and its typical dialogue function (clarification, reprise, direct question). For example, "what" sluices are often for clarification, while "who" sluices are frequently reprises.
- Heuristics are learnable: The core idea is that linguistic heuristics derived from corpus analysis can be formalized and then automatically learned by machine learning algorithms, achieving high accuracy. This bridges theoretical linguistics and practical NLP.
- Feature engineering is key: The features derived from the probabilistic Horn clauses proved to be highly predictive, suggesting that the linguistic principles captured by these features are well-motivated.
- Rule-based vs. Memory-based learning: Both SLIPPER (rule-based) and TiMBL (memory-based) performed comparably well, indicating that different ML paradigms can be effective for this task, with SLIPPER offering more interpretable rules.
- "Frag" feature is less useful: The feature indicating if the antecedent utterance was a fragment was found to have little predictive power, suggesting that other syntactic and semantic features are more critical for classification.
Key Results
- Both SLIPPER and TiMBL achieved high success rates of approximately 90% on the sluice classification task.
- Specifically, using a holdout method, SLIPPER achieved a success rate of 90%, with recall, precision, and f-measure values around 85-91% for the main categories (direct, reprise, clarification).
- TiMBL also achieved a success rate of 89%, with similar performance metrics.
- The agreement among the three expert annotators was moderate (Kappa = 52), but increased significantly when using only the two most experienced annotators (Kappa = 70-71), indicating that with proper training and experience, reliable annotation is achievable.
Limitations and Future Work
- Annotation agreement: While improved with expert annotators, the initial agreement was moderate, suggesting that some sluice categories are inherently ambiguous.
- Feature "frag": The feature indicating if the antecedent was a fragment was found to have low predictive power, suggesting potential for feature refinement.
- Future Work: The authors plan to develop an automatic procedure for annotating dialogue corpora with the features they used. This would enable the application of their machine learning algorithms to larger datasets without manual feature extraction, further advancing automatic dialogue processing.
Summary updated