[RETRACTED] A Contextual Hierarchical Attention Network with Adaptive Objective for Dialogue State Tracking

Yong Shan, Zekang Li, Jinchao Zhang, Fandong Meng, Yang Feng, Cheng Niu, Jie ZhouACL 2020

TL;DR

This paper introduces a novel "Contextual Hierarchical Attention Network" (CHAN) for dialogue state tracking (DST) that better leverages historical conversation context. It also proposes an "adaptive objective" to tackle the issue of slot imbalance, where some slots are much harder to track than others, leading to improved overall performance.

What Problem Does This Paper Address?

Dialogue state tracking (DST) is a crucial component of task-oriented dialogue systems, aiming to understand user intent by tracking slot-value pairs throughout a conversation. Traditional DST models often struggle to effectively utilize the entire dialogue history, focusing too much on the current turn. Additionally, many existing methods treat all slots equally, ignoring the "slot imbalance" problem where some slots appear far less frequently or are inherently harder to predict, leading to suboptimal performance on these "hard" slots. This paper aims to address both the limited use of historical context and the slot imbalance issue in DST.

Methods

The authors propose a two-pronged approach:

  • Contextual Hierarchical Attention Network (CHAN): This network is designed to capture relevant information at both word and turn levels from the dialogue history.
  • Slot-Word Attention: Initially, it identifies relevant words within utterances for each slot.
  • Context Encoder: This component models the contextual relationships between the word-level slot information across multiple turns.
  • Slot-Turn Attention: This further aggregates turn-level relevant information.
  • Global-Local Fusion Gate: A mechanism to dynamically balance information from the overall dialogue context and the current turn's utterance.
  • Adaptive Objective: To combat slot imbalance, this method dynamically adjusts the learning weights for each slot during training. It does this by evaluating the difficulty of each slot based on its accuracy on a validation set and also considers the confidence of individual sample predictions. This is a departure from static approaches like Focal Loss, aiming to adaptively focus on harder slots and samples.
  • State Transition Prediction: An auxiliary task is introduced to jointly train with DST. This task predicts whether a slot's value has changed from the previous turn, acting as an additional signal to improve context understanding.

Datasets

The paper evaluates its model on two large, widely-used datasets for task-oriented dialogue:

  • MultiWOZ 2.0: A multi-domain dataset with 7 domains and 35 slot-value pairs.
  • MultiWOZ 2.1: An improved version of MultiWOZ 2.0, which fixes annotation errors.

The authors use a subset of 5 domains from these datasets for their experiments, excluding "hospital" and "police" as they rarely appear in the test set.

Key Insights and Ideas

  • Hierarchical Attention is Key: The paper strongly suggests that effectively modeling dialogue history requires looking at context at multiple granularities (word and turn levels) and then fusing this information.
  • Slot Imbalance is a Real Problem: The authors provide clear evidence that the varying frequency and difficulty of slots significantly impacts overall DST performance, and addressing this is crucial for state-of-the-art results.
  • Adaptive Learning for Hard Slots: Dynamically adjusting learning weights based on slot difficulty and sample confidence is more effective than static methods for improving performance on challenging slots.
  • Auxiliary Tasks Help: The State Transition Prediction task acts as a beneficial regularization and context-enhancement mechanism for the primary DST task.

Key Results

The proposed model achieves state-of-the-art results on both MultiWOZ 2.0 and MultiWOZ 2.1:

  • MultiWOZ 2.0: Achieved 52.68% joint accuracy, a significant improvement of +1.24% over previous best results.
  • MultiWOZ 2.1: Achieved 58.55% joint accuracy, a substantial improvement of +5.98% over previous best results.
  • Slot accuracy also saw improvements, reaching 97.69% on MultiWOZ 2.0 and 98.14% on MultiWOZ 2.1.
  • Ablation studies showed that both the State Transition Prediction task and the Adaptive Objective contributed positively to the overall performance, with the adaptive objective being particularly important for improving accuracy on difficult slots.

Limitations and Future Work

  • Ontology Dependency: While the model is designed to be scalable to unseen domains, slots, and values, the core CHAN architecture is presented in the context of a predefined ontology. The authors mention that they will explore applying these techniques to open-vocabulary models in the future.
  • Retraction: The paper has been retracted, indicating potential issues with the research or its presentation that are not detailed in the provided text. This is a significant limitation that supersedes any other.

Summary updated

BibTeX
@inproceedings{shan2020retracted,
  author = {Yong Shan and Zekang Li and Jinchao Zhang and Fandong Meng and Yang Feng and Cheng Niu and Jie Zhou},
  title = {{[RETRACTED] A Contextual Hierarchical Attention Network with Adaptive Objective for Dialogue State Tracking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure