Automatic Recognition of the General-Purpose Communicative Functions Defined by the ISO 24617-2 Standard for Dialog Act Annotation (Extended Abstract)

Eugénio Ribeiro, Ricardo Ribeiro, David Martins de MatosIJCAI 2023 · DOI

TL;DR

This paper tackles the challenge of automatically recognizing "general-purpose communicative functions" in dialogues, which are organized hierarchically according to the ISO 24617-2 standard. The authors propose a novel end-to-end hierarchical network that leverages transfer learning to overcome data scarcity, achieving state-of-the-art results on the DialogBank dataset.

What Problem Does This Paper Address?

Understanding the intention behind utterances in a dialogue is crucial for building effective dialogue systems. Traditional dialogue act recognition focuses on assigning a single label to each utterance. However, the ISO 24617-2 standard introduces a more complex, hierarchical structure for communicative functions, where functions can have sub-functions and the hierarchy doesn't always end at a "leaf" node. This hierarchical nature, combined with the limited availability of data annotated with this standard, makes automatic recognition a significant challenge. Existing approaches often flatten this hierarchy or use multiple independent classifiers, failing to fully leverage the structural information.

Methods

The authors adapt existing dialog act recognition approaches to handle the hierarchical classification problem. Their core innovation is an end-to-end hierarchical network that predicts communicative functions at multiple levels of the hierarchy simultaneously. Key components of their approach include:

  • Cascading Outputs: The probability distribution predicted at one level of the hierarchy is fed as input to the next level, allowing the network to learn dependencies between functions at different levels.
  • Specialization Layers: Each level of the hierarchy has a dedicated "specialization layer" that learns to extract the most relevant features for distinguishing functions at that specific level.
  • Variable-Length Path Prediction: To handle the fact that not all paths end at a leaf node, they introduce a "None" label at each level, allowing for flexible path termination.
  • Maximum A Posteriori (MAP) Estimation: Instead of a greedy top-down prediction, they use MAP estimation to find the most probable valid path through the hierarchy, mitigating the impact of errors in earlier levels.
  • Transfer Learning: To address the data scarcity issue, they leverage pre-trained dialog act recognition models, fine-tuning them to capture general intention information and then focusing on the specific hierarchical structure.

Datasets

The primary dataset used for training and evaluation is the DialogBank. This dataset contains dialogues annotated according to the ISO 24617-2 standard. The authors specifically focus on the 15 English dialogues within DialogBank, comprising 2,360 annotated segments. They note that the distribution of general-purpose communicative functions in DialogBank is highly unbalanced, with "Inform" being the most frequent.

Key Insights and Ideas

  • Hierarchy Matters: The results strongly suggest that treating communicative functions as a flat classification problem is suboptimal. The hierarchical approach, by design, captures structural dependencies that are crucial for accurate recognition.
  • End-to-End is Powerful: A single end-to-end hierarchical network outperforms multi-classifier approaches, indicating that joint learning of all levels is more effective than piecemeal classification.
  • Transfer Learning is Essential: Given the limited annotated data for the ISO 24617-2 standard, transfer learning from models trained on other dialog act corpora significantly boosts performance.
  • "None" Label is Key: The introduction of a "None" label at each level is a clever way to handle variable-length hierarchical paths without overly complicating the network architecture.
  • MAP Estimation Improves Robustness: Using MAP estimation for path selection is more robust to errors in higher levels of the hierarchy compared to a simple greedy top-down approach.

Key Results

  • The proposed E2E hierarchical approach achieved the highest F-measure (74.16%) when considering all segments, outperforming both flat and multi-classifier hierarchical baselines.
  • While the flat approach achieved a slightly higher exact match ratio (74.17%), the hierarchical approach demonstrated better recall, indicating its ability to identify less prominent or deeper functions.
  • When evaluating only on segments with communicative functions in the Task dimension (Table 2), the E2E hierarchical approach outperformed the flat approach across all metrics, confirming its suitability for this specific task.
  • Ablation studies revealed that per-level specialization layers had a greater impact on performance than cascading outputs, and transfer learning was crucial.

Limitations and Future Work

  • Data Scarcity: Despite transfer learning, the small size of the DialogBank dataset limits the model's ability to fully capture the nuances of deeper hierarchical functions, leading to a gap between precision and recall.
  • Unbalanced Data: The inherent imbalance in the DialogBank dataset poses a challenge, and further work could explore more advanced techniques for handling class imbalance.
  • Beyond General-Purpose Functions: The paper focuses on general-purpose communicative functions. Future work should address the automatic recognition of dimension-specific functions and the complete dialog act annotation, which requires more annotated data.
  • Increasing Annotated Data: The authors emphasize the need for more annotated dialogs following the ISO 24617-2 standard to further advance research in this area.

Summary updated

BibTeX
@inproceedings{ribeiro2023automatic,
  author = {Eugénio Ribeiro and Ricardo Ribeiro and David Martins de Matos},
  title = {{Automatic Recognition of the General-Purpose Communicative Functions Defined by the ISO 24617-2 Standard for Dialog Act Annotation (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/788},
}
Figure