A Multi-Dimensional, Cross-Domain and Hierarchy-Aware Neural Architecture for ISO-Standard Dialogue Act Tagging

Stefano Mezza, Wayne Wobcke, Alan BlairCOLING 2022

TL;DR

This paper introduces DASHNet, a novel neural architecture for Dialogue Act (DA) tagging that leverages the hierarchical structure of the ISO 24617-2 standard, syntactic information, and contextual cues. It achieves state-of-the-art results on challenging benchmarks, demonstrating improved generalization across domains, particularly for low-frequency DA tags.

What Problem Does This Paper Address?

Dialogue Act (DA) tagging, the process of identifying the communicative function of an utterance, is crucial for conversational AI systems. However, accurately tagging dialogue acts is challenging due to the complexity of existing standards like ISO 24617-2, which is multi-dimensional (e.g., Task, Social, Feedback), hierarchical, and covers a wide range of functions. A major hurdle is the scarcity of large, consistently annotated datasets for this standard, making it difficult to train robust models, especially for fine-grained tags. Existing approaches often struggle with the multi-dimensional and hierarchical nature of the standard, and models trained on specific domains tend to perform poorly on unseen conversational data.

Methods

The authors propose DASHNet (Dialogue Act Syntax and Hierarchy-aware Network), a neural architecture designed to address these challenges. DASHNet employs a triple input encoding mechanism:

  • Lexical Encoder: Processes input utterances using pre-trained GloVe embeddings and a bidirectional GRU to capture semantic information.
  • Syntax Encoder: Encodes Part-of-Speech (POS) and Dependency (DEP) tags using linear layers and a bidirectional GRU, providing syntactic context.
  • Context Encoder: Incorporates information from the previous speaker's utterance to capture contextual dependencies.

Crucially, DASHNet integrates prior knowledge about the hierarchical relationships between DA tags. This is achieved by estimating prior probabilities between parent and child nodes in the DA taxonomy and encoding this information using a Bidirectional Tree-LSTM. This allows the model to better understand how different DA tags relate to each other, improving accuracy, especially for less frequent tags.

Datasets

The authors created a General Dialogue Corpus (GDC) by aggregating and mapping several existing datasets to their subset of the ISO 24617-2 taxonomy. The GDC includes:

  • Switchboard Dialog Act Corpus (SWDA): Focuses on information exchange.
  • ICSI Meeting Recorder Dialog Act (MRDA) Corpus: Features academic meetings with conversational structuring tags.
  • DailyDialog: A human-written corpus of everyday conversations with a balance of information transfer and action-discussion tags.
  • Mastodon Corpus: A Twitter-like corpus with a mix of information exchange and persuasive dialogue, including internet slang.
  • Internet Argument Corpus v2 (IAC) (4Forums subset): Contains argumentative dialogue with agreement/disagreement annotations.

The primary evaluation benchmark used is the DialogBank corpus, which is manually annotated with the ISO 24617-2 standard, making it a valuable resource for out-of-domain testing.

Key Insights and Ideas

  • Hierarchy Matters for Generalization: Incorporating the hierarchical structure of DA tags significantly improves the model's ability to generalize to out-of-domain data. This suggests that understanding the relationships between tags is more important for cross-domain performance than for in-domain tasks.
  • Syntax Aids Low-Frequency Tags: Grammatical features (POS and dependency tags) have a marginal impact on overall accuracy (Micro-F1) but a substantial positive effect on the classification of low-frequency DA tags (Macro-F1). This indicates their utility in distinguishing subtle or rare communicative functions.
  • Context is King (for some tags): Contextual features from the previous utterance provide the biggest overall boost to performance, both in-domain and out-of-domain, aligning with previous research highlighting the importance of context.
  • Corpus Diversity is Key: Each corpus in the GDC contributes to the model's performance, with larger corpora like SWDA and MRDA having a significant impact. The Mastodon corpus, despite its size, proved surprisingly effective due to its ISO-aligned annotation scheme.

Key Results

DASHNet achieved state-of-the-art results on the DialogBank test set, outperforming previous models like HiAGM-TP and SVM classifiers.

  • On the DialogBank test set, DASHNet achieved a Micro-F1 of 83.7% and a Macro-F1 of 57.1%.
  • On the GDC test set, DASHNet achieved a Micro-F1 of 90.6% and a Macro-F1 of 76.9%.
  • Ablation studies showed that contextual features provided the largest performance boost, followed by the hierarchical Tree-LSTM component. Grammatical features were most beneficial for low-frequency tags.

Limitations and Future Work

  • Partial Mappings: The authors acknowledge that their mappings from existing corpora to the ISO standard are partial, meaning some utterances could not be accurately annotated. These were handled by using coarse-grained tags or discarding them.
  • Limited Context: The context encoder was limited to the previous utterance. Future work could explore incorporating a wider context of dialogue and DA history.
  • Future Directions: The authors plan to expand their work by covering more ISO-annotated corpora, extending the context encoder, and experimenting with different embedding mechanisms like BERT and RoBERTa. They also aim to make their code and mapped dataset publicly available to foster further research.

Summary updated

BibTeX
@inproceedings{mezza2022multi,
  author = {Stefano Mezza and Wayne Wobcke and Alan Blair},
  title = {{A Multi-Dimensional, Cross-Domain and Hierarchy-Aware Neural Architecture for ISO-Standard Dialogue Act Tagging}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
}
Figure