Abstract
Dialogue State Tracking (DST) is a core component of task-oriented dialogue systems, responsible for maintaining an accurate representation of user goals throughout a conversation. Traditional models often treat multi-turn dialogues as flat sequences, leading to a "needle in a haystack" problem as history grows. This paper proposes the Hierarchical Dynamic Copy Network (HDCN), which employs a two-level hierarchical attention mechanism and a novel "focus loss" to pinpoint relevant information turns. HDCN achieves a joint accuracy of 46.76% on the MultiWOZ 2.1 dataset, outperforming existing non-pretrained models while providing superior interpretability.
Introduction
In task-oriented dialogue, the system must track specific slot-value pairs (e.g., "hotel-area: east") across multiple turns. Recent state-of-the-art models like TRADE treat the entire dialogue history as a single long string. However, as dialogues lengthen, decoders struggle to copy the correct values from massive sequences. Standard hierarchical models often fail because they use static turn representations that may discard nuances necessary for specific slots. The authors argue that an effective DST model must "learn to focus" on the specific turn where relevant information is introduced or modified.
Methodology
The HDCN architecture introduces three primary innovations to the standard encoder-decoder framework:
1. Hierarchical Dynamic Attention
Instead of a flat attention pass, HDCN operates at two levels:
- Word-level attention: Identifies important tokens within a specific turn.
- Turn-level attention: Determines which turn in the dialogue history is most relevant to the current slot being decoded.
Crucially, turn representations are dynamic. They are calculated on-the-fly during decoding based on word-level attention, allowing a single turn to be represented differently depending on whether the model is searching for a "hotel name" or a "booking day."
2. Turn Focus Mechanism
The authors define an "information turn" as the specific turn where a slot’s value is first mentioned or changed. To guide the model, they introduce a Focus Loss term. This supervised attention mechanism penalizes the model if it fails to assign the highest turn-level attention to the ground-truth information turn during training.
3. HDCN-cover
To handle multi-word values (e.g., "holy trinity church"), the model incorporates a "coverage mechanism." While typically used in summarization to prevent repetition, here it is adapted to encourage the model to maintain focus on the same turn while generating multi-token values.
Experimental Setup
The model was evaluated on MultiWOZ 2.1, a refined version of the largest multi-domain dialogue dataset. The 2.1 version was chosen to mitigate the ~32% annotation error rate found in the original dataset. Since the dataset lacks explicit "information turn" labels, the authors utilized a rule-based heuristic to infer them for training the Focus Loss.
Key Results and Analysis
HDCN demonstrates significant improvements over previous non-pretrained architectures. The best-performing variant, HDCN-cover with last-initialization, achieved a joint accuracy of 46.76%, representing a 1.16% absolute improvement over the TRADE model.
Interpretability and Initialization
Beyond accuracy, the hierarchical approach yields cleaner, more explainable attention maps. The model effectively uses a "sentry turn" (a prepended "none" token) to handle slots that have not yet been mentioned. Furthermore, the authors found that "last-initialization"—initializing a turn's hidden state with the final state of the previous turn—outperformed zero-initialization by approximately 1%, highlighting the importance of preserving context across turns.
Conclusion
While HDCN does not currently surpass massive pretrained models like TRIPPY, it establishes a new benchmark for lightweight, non-pretrained DST models. By moving from flat sequences to a hierarchical, focus-driven architecture, HDCN offers a more practical and interpretable solution for real-time dialogue systems. Future work may involve integrating these hierarchical focus mechanisms into large-scale pretrained Transformers to combine structural inductive biases with broad linguistic knowledge.
Summary updated