Improving the Quality of Minority Class Identification in Dialog Act Tagging

Adinoyi Omuya, Vinodkumar Prabhakaran, Owen RambowNAACL 2013

TL;DR

This paper tackles the challenge of accurately identifying rare "minority" dialog acts in email conversations. By optimizing features for each dialog act classifier individually and then using a cascading approach to combine their predictions, the authors significantly reduce errors for these underrepresented classes, leading to better overall dialog act tagging performance.

What Problem Does This Paper Address?

Dialog act (DA) tagging aims to categorize the pragmatic meaning of an utterance within a conversation. While standard multi-class classification methods, like Support Vector Machines (SVMs), perform reasonably well on common dialog acts, they struggle with "minority" classes – those that appear infrequently in the training data. This is a common problem in many classification tasks where imbalanced datasets lead to poor performance on underrepresented categories. The authors focus on email conversations, where a specific DA, REQUEST-ACTION (e.g., "write a report"), is rare and thus difficult to classify accurately using traditional methods. Existing approaches often treat all classes equally, leading to a bias towards the majority classes and poor performance on minority ones.

Methods

The paper introduces a "Divide and Conquer" (DAC) approach that builds upon a standard multi-class SVM framework. The core innovations are:

  • Per-Class Feature Optimization: Instead of optimizing features for the entire multi-class classifier at once, the DAC system optimizes features separately for each individual binary classifier (one for each dialog act). This allows each classifier to focus on the features most relevant to its specific class, especially beneficial for minority classes.
  • Cascading Classifier Combination: Rather than relying solely on the classifier with the highest confidence score (the standard "one-against-all" approach), the DAC system employs a cascade. Classifiers are applied in ascending order of their class frequency in the training data (least frequent first). The first classifier to positively identify the dialog act determines the final prediction. If no classifier in the cascade predicts positively, a confidence-based method is used as a fallback.

Two variants of the DAC system are explored:

  • Basic DAC: Uses per-class feature optimization and a confidence-based combination.
  • Minority Preference DAC (DACMP): Prioritizes a specific minority class (REQUEST-ACTION) by checking its classifier first. If it predicts true, that class is chosen; otherwise, it falls back to the basic DAC system.
  • Cascading Minority Preference DAC (DACCMP): Extends DACMP by using an ordered list of minority classes (based on frequency) in a cascading manner.

Datasets

The study uses the Enron email corpus, specifically a subset of 122 email threads (360 messages, 20,740 word tokens) that was manually annotated for dialog acts by Hu et al. (2009). The DA labels considered are REQUEST-INFORMATION (R-I), REQUEST-ACTION (R-A), CONVENTIONAL (CONV), and INFORM (INF). The paper notes that other DA labels occurring less than 5 times were ignored.

Key Insights and Ideas

  • Minority classes are a significant challenge: The paper clearly demonstrates that standard classification approaches struggle with rare dialog acts, leading to poor performance on these crucial categories.
  • Per-class optimization is key: Optimizing features for each classifier individually, rather than globally, is crucial for boosting performance on minority classes. This allows the model to learn more nuanced patterns specific to each DA.
  • Cascading classifiers offer a robust alternative: The cascading approach, prioritizing less frequent classes, proves more effective than simple confidence-based selection, especially when dealing with imbalanced data. It prevents minority classes from being "drowned out" by majority classes.
  • Feature importance varies by DA: The post-hoc analysis reveals that different DA categories rely on different types of features (e.g., POS n-grams for REQUEST-ACTION vs. word n-grams for INFORM/CONVENTIONAL), reinforcing the benefit of per-class optimization.

Key Results

The proposed methods significantly improve dialog act tagging performance, particularly for the minority class REQUEST-ACTION:

  • Basic DAC: Achieved a 15.6% F-measure error reduction for REQUEST-ACTION over the baseline system.
  • DACMP: Further improved the F-measure error reduction for REQUEST-ACTION to 22.8%.
  • DACCMP: Achieved the best overall results, with a 22.8% F-measure error reduction for the minority class REQUEST-ACTION, and an overall accuracy error reduction of 10.6% compared to the baseline. It also improved performance for the other three classes.
  • Extrinsic Evaluation: When used to predict dialog acts for an Overt Displays of Power (ODP) task, the DACCMP system (using predicted DA tags) achieved an F-measure of 54.2, which was a 12.1% error reduction compared to using the baseline system's predicted DA tags (F=52.6), and a 26.6% error reduction compared to a system that didn't use DA features at all (F=50.0).

Limitations and Future Work

  • Limited to specific DAs: The study focuses on four main dialog act categories and ignores very rare ones. Future work could explore methods to handle extremely infrequent classes.
  • Feature set limitations: While the feature engineering is extensive, the paper acknowledges that the best performing feature sets for each configuration are not reported due to space constraints. Further exploration of feature combinations could yield additional improvements.
  • Extrinsic evaluation scope: The extrinsic evaluation on the ODP task shows promise, but further testing on other downstream tasks would be beneficial to demonstrate the broader applicability of the improved DA tagging.
  • Computational cost: While not explicitly detailed, per-class optimization and cascading might introduce higher computational costs compared to a single multi-class classifier, which could be a consideration for real-time applications.

Summary updated

BibTeX
@inproceedings{omuya2013improving,
  author = {Adinoyi Omuya and Vinodkumar Prabhakaran and Owen Rambow},
  title = {{Improving the Quality of Minority Class Identification in Dialog Act Tagging}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure