Dialogue State Tracking with Sparse Local Slot Attention

Longfei Yang, Jiyi Li, Sheng Li, Takahiro ShinozakiACL 2023

TL;DR

This paper introduces Sparse Local Slot Attention (SLSA), a novel mechanism for dialogue state tracking (DST) that focuses attention on relevant local segments of dialogue history. By employing sparse attention and local context, SLSA improves performance, especially for name-related slots and longer dialogues, outperforming traditional dense token-wise attention methods.

What Problem Does This Paper Address?

Task-oriented dialogue systems rely heavily on Dialogue State Tracking (DST) to understand and fulfill user requests. DST models aim to track the dialogue state, typically represented as a list of (domain, slot, value) pairs, which guides the system's subsequent actions. A critical challenge in DST is effectively extracting relevant information from the dialogue history, which can be noisy and lengthy. Existing mainstream models often use a "fully token-wise slot attention" mechanism. While this approach has shown promise, it has drawbacks: it can disperse attention, leading to the model overlooking crucial local relationships between words or phrases. Furthermore, the standard softmax function used in attention can assign non-zero probabilities to irrelevant parts of the dialogue history, potentially confusing the model, especially as dialogues get longer. This paper addresses the need for a more focused and efficient attention mechanism that can better capture local semantic dependencies relevant to specific slots.

Methods

The proposed approach, Sparse Local Slot Attention (SLSA), introduces two key innovations:

  • Sparse Local Slot Attention: Instead of attending to every token in the dialogue history, SLSA first extracts "local semantic information" using a convolutional layer with a defined window size and stride. This captures local dependencies for each slot. Then, a "sparse attention" mechanism, utilizing the sparsemax function, is applied. Sparsemax is designed to produce sparse probability distributions, effectively assigning zero probability to irrelevant parts of the dialogue history and forcing the model to focus on more pertinent segments. This contrasts with the dense, full-support distributions typically produced by softmax.
  • BERT-based Encoding: The model leverages pre-trained BERT models for encoding both the dialogue context (BERTcontext) and the slots/candidate values (BERTsv). The BERTcontext parameters are fine-tuned during training, while BERTsv parameters are frozen.

The overall framework involves encoding the dialogue history and slots, applying the sparse local slot attention to extract slot-specific representations, and then using a slot self-attention mechanism to allow communication between slot representations. Finally, slot value matching is performed using Euclidean distance to predict the state value.

Datasets

The paper evaluates its approach on two widely used datasets for task-oriented dialogue:

  • MultiWOZ 2.0: A large-scale, multi-domain human-human conversational dataset.
  • MultiWOZ 2.4: A later version of MultiWOZ that includes corrections for annotations, aiming to provide a cleaner benchmark.

Key Insights and Ideas

  • Local Context is Crucial: The paper strongly suggests that focusing on local contextual information, rather than just individual tokens, is vital for accurate DST, especially for complex entities like names.
  • Sparsity Improves Focus: The use of sparse attention (via sparsemax) effectively prunes irrelevant information, allowing the model to concentrate on the most informative parts of the dialogue history for each slot.
  • Name-Related Slots Benefit Most: The proposed SLSA mechanism shows particular effectiveness in predicting values for name-related slots (e.g., attraction names, restaurant names), which are often informal and locally compact.
  • Long Dialogues are a Differentiator: SLSA demonstrates a more robust performance on longer dialogues compared to previous state-of-the-art models, indicating its ability to manage context accumulation and avoid performance degradation.

Key Results

  • On the MultiWOZ 2.0 dataset, the proposed SLSA model achieved a Joint Goal Accuracy (JGA) of 54.83%, slightly outperforming the previous state-of-the-art STAR model (54.53%).
  • On the MultiWOZ 2.4 dataset, SLSA achieved a JGA of 77.92%, showing a significant improvement over previous models.
  • An ablation study confirmed that both the "sparse" and "local" components of the attention mechanism are crucial for performance. Removing either component led to a decrease in JGA.
  • Qualitative analysis showed that SLSA could better focus on relevant local entities and correctly interpret user intentions (e.g., "nothing in particular" for an area preference) where other models struggled.

Limitations and Future Work

The authors acknowledge the following limitations:

  • Difficulty with Generation-Based DST: Applying SLSA to generation-based DST tasks might be challenging, as generative models may require more continuous and successive information for language generation, making it harder to isolate "local" or "sparse" properties.
  • Fixed Convolutional Window: The current convolutional operation considers a fixed-size local context. Handling local properties of varying lengths remains a challenge.

Future work could explore how to adapt SLSA for generative DST and investigate more flexible methods for capturing local context that can adapt to different lengths.

Summary updated

BibTeX
@inproceedings{yang2023dialogue,
  author = {Longfei Yang and Jiyi Li and Sheng Li and Takahiro Shinozaki},
  title = {{Dialogue State Tracking with Sparse Local Slot Attention}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure