SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking

Hwaran Lee, Jinsik Lee, Tae-Yoon Kim

Introduction

In goal-oriented dialog systems, Belief Tracking (or Dialog State Tracking) is the critical process of maintaining a record of a user's requirements at every turn. For instance, a system must track that a user is searching for a "cheap" "Italian" restaurant in the "center" of town. Traditionally, these systems have been "brittle," relying on slot-dependent architectures that require redesigning output layers and retraining from scratch whenever a new domain (e.g., "taxi") or slot (e.g., "pickup time") is introduced. Furthermore, traditional models often struggle with lexical variations, failing to recognize that "inexpensive" and "cheap" refer to the same concept.

SUMBT (Slot-Utterance Matching Belief Tracker) addresses these challenges by treating belief tracking as a semantic matching problem rather than a fixed classification task. By leveraging pre-trained language models, SUMBT creates a universal and scalable framework capable of handling any domain or slot type.

Methodology

The SUMBT architecture is built upon four primary components designed to maximize semantic understanding and scalability:

  1. Contextual Semantic Encoders: The model utilizes BERT to encode inputs. Unlike previous methods, it encodes not only the user and system utterances but also the literal names of the slots (e.g., "restaurant-food") and candidate values (e.g., "Italian"). This allows the model to inherit BERT’s deep pre-trained knowledge of language semantics.
  2. Slot-Utterance Matching: Inspired by machine reading comprehension, the model employs Multi-head Attention. It treats the encoded slot-type as a "query" to extract relevant information from the utterance "context."
  3. Belief Tracker: To maintain state across a conversation, the attended context is processed by a Recurrent Neural Network (RNN), such as a GRU or LSTM, ensuring the model retains constraints mentioned in previous turns.
  4. Non-parametric Discriminator: Instead of a standard softmax layer with fixed outputs, SUMBT uses metric learning. It calculates the distance (Euclidean or Cosine) between the tracker’s output and the BERT-encoded representations of all possible candidate values. The value with the minimum distance is selected.
The SUMBT architecture, featuring BERT encoders for slots and utterances, multi-head attention for matching, and a non-parametric discriminator.

Evaluation

The authors evaluated SUMBT on two major benchmarks:

  • WOZ 2.0: A single-domain dataset focused on restaurant reservations with three slots (area, food, price range).
  • MultiWOZ: A complex multi-domain dataset (7 domains, 35 slots) that serves as the standard for testing scalability in dialog systems.

Key Results and Insights

SUMBT achieved state-of-the-art (SOTA) performance at the time of its publication. On WOZ 2.0, it reached 91.0% joint accuracy, outperforming previous models like GLAD (88.1%) and StateNetPSI (88.9%). On the more challenging MultiWOZ dataset, it achieved 42.4% joint accuracy, a significant improvement over the benchmark baseline of 25.8%.

Semantic Universality and Scalability

By encoding slot and value names, the model treats "price range" as a semantic concept rather than a simple index. This allows the model to understand that "cheap" relates to "price" based on its pre-training. Because the discriminator is non-parametric, new candidate values can be added to the ontology and encoded with BERT, allowing the model to match them immediately without retraining.

Qualitative Success

Attention visualizations confirm that the model correctly identifies relevant phrases. For example, when tracking the "price range" slot, the model's attention weights spike on phrases like "reasonably priced" within the user's utterance.

Attention visualization across three dialog turns, demonstrating how the model focuses on relevant tokens (e.g., 'reasonably priced' for 'price range') to update the belief state.

Limitations and Future Work

While SUMBT is highly scalable, it remains dependent on a predefined ontology (a list of candidate values) during inference. It cannot currently generate values not present in its list, such as specific phone numbers or addresses. Additionally, the use of BERT for every slot at every turn increases computational costs compared to simpler RNN-based models. Future research may explore continual learning to allow the model to incorporate new domains and slots dynamically as a system evolves.

Summary updated

Abstract

In goal-oriented dialog systems, belief trackers estimate the probability distribution of slot-values at every dialog turn. Previous neural approaches have modeled domain- and slot-dependent belief trackers, and have difficulty in adding new slot-values, resulting in lack of flexibility of domain ontology configurations. In this paper, we propose a new approach to universal and scalable belief tracker, called slot-utterance matching belief tracker (SUMBT). The model learns the relations between domain-slot-types and slot-values appearing in utterances through attention mechanisms based on contextual semantic vectors. Furthermore, the model predicts slot-value labels in a non-parametric way. From our experiments on two dialog corpora, WOZ 2.0 and MultiWOZ, the proposed model showed performance improvement in comparison with slot-dependent methods and achieved the state-of-the-art joint accuracy.
BibTeX
@inproceedings{lee2019sumbt,
  author = {Hwaran Lee and Jinsik Lee and Tae-Yoon Kim},
  title = {{SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1546},
}
Figure