Active learning for dialogue act classification

Björn Gambäck, Fredrik Olsson, Oscar TäckströmInterspeech 2011

TL;DR

This paper demonstrates that active learning significantly improves dialogue act classification accuracy while drastically reducing the amount of labeled data required. By intelligently selecting which data points to label, active learning achieves comparable or better performance than passive learning with a fraction of the annotation effort, even outperforming the state-of-the-art on existing benchmarks.

What Problem Does This Paper Address?

Classifying dialogue acts – the communicative intent behind utterances in a conversation – is crucial for building intelligent dialogue systems. Traditional supervised machine learning approaches require large amounts of manually labeled data, which is time-consuming and expensive to obtain. This paper addresses the challenge of building accurate dialogue act classifiers more efficiently by exploring the effectiveness of active learning techniques in this domain. The authors aim to show that active learning can significantly reduce the annotation burden without sacrificing, and potentially even improving, classification performance.

Methods

The core of this paper lies in applying and evaluating active learning for dialogue act classification. Unlike passive learning, where data is randomly sampled for labeling, active learning allows the learner to query an "oracle" (in this case, a simulated human annotator) for labels of the most informative instances. The authors use a Support Vector Machine (SVM) as the base learner, specifically an L2-regularized linear SVM implemented in LIBLINEAR.

The active learning strategy employed is batch-based, meaning small sets of instances are added to the training data at each iteration, rather than single instances. The selection criterion for these instances is the minimal-distance-to-hyperplane heuristic, which prioritizes instances closest to the decision boundaries, as these are considered the most uncertain for the current model. Features used include word n-grams (uni-, bi-, trigrams) of the current utterance, as well as features from the previous utterance to model dialogue context.

Datasets

The experiments were conducted on two distinct dialogue corpora:

  • Switchboard corpus: A large, English, human-human telephone speech corpus containing approximately 224,000 utterances from 1,155 conversations. This corpus uses the SWBD-DAMSL tagset, with the experiments utilizing all 44 dialogue act classes without merging or filtering.
  • Dihana corpus: A Spanish, human-machine (wizard-of-Oz) telephone dialogue corpus focused on train timetable information. It comprises 9,179 user and 13,829 wizard utterances, with a dialogue act annotation scheme based on three levels of information. The experiments primarily used Level 1 and 2 labels, with a total of 72 labels.

Key Insights and Ideas

  • Active learning is demonstrably superior to passive learning for dialogue act classification: The paper provides strong empirical evidence that active learning consistently outperforms passive learning in terms of accuracy, especially when considering the amount of labeled data used.
  • Significant reduction in annotation effort: The most compelling finding is the drastic reduction in the number of labeled instances needed. For the Dihana corpus, active learning achieved the same performance as passive learning using only 20% of the data, a five-fold reduction.
  • Active learning can improve upon the state-of-the-art: The passive SVM baseline itself sets a new state-of-the-art on the Switchboard corpus, and active learning further enhances this performance.
  • Contextual features are vital: The inclusion of features from previous utterances significantly boosts classification accuracy, highlighting the importance of dialogue history.
  • Batch-based active learning is effective but might be slightly sub-optimal: While batch-based learning speeds up the process, instance-based selection could potentially yield even higher accuracy, though at the cost of more iterations.

Key Results

  • Switchboard Corpus:
    • Active learning achieved an error reduction of approximately 7% compared to passive learning.
  • The passive SVM baseline achieved a peak accuracy of 77.85% (±0.26) on the 44-class tagset, surpassing previous state-of-the-art results by 31.06% in error reduction.
  • Dihana Corpus:
    • Active learning achieved a factor of 5 reduction in the amount of labeled data needed to reach the maximum passive learning performance.
  • The passive learner achieved 91.61% (±0.60) accuracy on Level 1 & 2 labels with a [-1, 0] context, and 94.08% (±0.36) with a [-3, 0] context.
  • Active learning reached the maximum passive learning level after using only 20% of the data.

Limitations and Future Work

  • Simulated Oracle: The experiments used a simulated oracle (pre-labeled data) rather than a real human annotator. While this ensures consistency and faster experiments, it might not fully capture the complexities and potential biases of human annotation in real-time.
  • Batch vs. Instance-Based Learning: While batch-based learning was used for efficiency, the paper notes that instance-based learning might yield slightly better results, suggesting further investigation into optimal batch sizes and selection strategies.
  • Structured Learning: The authors suggest exploring structured prediction models, particularly for more deterministic corpora like Dihana, as initial experiments showed promise for structured SVMs in improving accuracy.
  • Hyperparameter Optimization: The paper mentions that hyperparameter optimization (specifically the SVM cost parameter 'c') was not fully explored but indicates that it could further improve results, with lower 'c' values potentially leading to better classification and faster training.

Summary updated

BibTeX
@inproceedings{gamback2011active,
  author = {Björn Gambäck and Fredrik Olsson and Oscar Täckström},
  title = {{Active learning for dialogue act classification}},
  booktitle = {Interspeech},
  year = {2011},
}
Figure