MoPE: Mixture of Prefix Experts for Zero-Shot Dialogue State Tracking

Tianwen Tang, Tong Zhu, Haodong Liu, Yin Bai, Jia Cheng, Wenliang Chen

TL;DR

MoPE (Mixture of Prefix Experts) is a novel parameter-efficient framework that significantly boosts zero-shot Dialogue State Tracking (DST) by clustering similar slots across different domains. By training specialized prefix prompts for each cluster, MoPE allows a smaller language model (<10B parameters) to outperform massive LLMs like ChatGPT and Codex on standard DST benchmarks.

What Problem Does This Paper Address?

Task-oriented dialogue (TOD) systems rely on Dialogue State Tracking (DST) to understand user intentions by extracting information into domain-slot-value triplets (e.g., hotel-area-centre). However, training a DST model for every new domain requires massive amounts of labeled data.

While zero-shot DST aims to solve this by transferring knowledge to unseen domains, existing models face two major hurdles:

  1. Domain Transfer: Models struggle to generalize to entirely new domains because they don't explicitly map connections between seen and unseen concepts.
  2. Partial-Prediction: When encountering a new domain, models frequently fail to predict all the necessary slot values, leading to incomplete dialogue states.

The authors observe that while domains change, the underlying slots often share semantic meanings and value formats. For example, "attraction area," "restaurant area," and "hotel area" all expect a location value like "centre." MoPE leverages these cross-domain similarities to bridge the zero-shot gap.

Figure 1: Illustration of dialogues in different do- mains share similar slot names even the same slot value.

Methods

MoPE introduces a three-step pipeline to solve the zero-shot DST task:

  1. Slot Clustering: Instead of treating slots independently, MoPE uses an unsupervised $k$-means algorithm to group semantically similar slots into $K$ distinct clusters. The features for clustering are derived from the hidden outputs of a pre-trained LLM.
  2. Deep Prefix Prompt Tuning: To keep training efficient and avoid full-model fine-tuning for every single cluster, the authors use Parameter-Efficient Transfer Learning (PETL). They initialize $K$ separate prefix prompt models (one per cluster). During training, the backbone LLM parameters remain frozen, and only the prefix prompts are optimized.
  3. Generation & Optimization: During inference, when an unseen slot is encountered, the framework maps it to the nearest cluster centroid. The corresponding specialized prefix prompt is then prepended to the LLM layers to guide the autoregressive generation of the slot value.
Figure 2: Illustration of our proposed method, including (a) Slot clustering, (b) Deep Prefix Prompt Tuning, and (c) Multiple Prefix Prompt Generation. Slot clustering is used to categorize all slots into distinct clusters and establishes connections between slots in different domains. Deep Prefix Prompt Tuning is our method to strengthen the LLM’s conditional generation. Multiple Prefix Prompt Generation shows the complete pipeline of solving DST task.

Datasets

The authors evaluated MoPE on two widely recognized multi-domain dialogue benchmarks:

  • MultiWOZ 2.1: A human-to-human dataset covering multiple domains. Following standard zero-shot protocols, the model is evaluated on five specific domains (Attraction, Hotel, Restaurant, Taxi, Train).
  • SGD (Schema-Guided Dialogue): A massive machine-to-machine dataset containing over 16,000 dialogues across 16 domains, explicitly featuring unseen domains in the test set to rigorously evaluate zero-shot capabilities.

Key Insights and Ideas

  • Hidden Outputs Beat Word Embeddings: When clustering slots, using the LLM's contextual hidden outputs yields significantly better tracking accuracy than using raw word embeddings. Hidden outputs capture deeper semantic relationships, helping group similar slots more effectively.
Figure 3: Zero-shot results on the attraction domain with different representations of slot feature.
  • Specialized Experts Reduce Omissions: The error analysis reveals that MoPE drastically reduces "partial-prediction" errors compared to standard deep prefix prompt tuning (DPPT). By routing a slot to a specialized expert, the LLM becomes highly sensitive to the exact context needed for that specific type of slot.
Figure 4: The slot error distribution of MoPE and DPPT.

Key Results

MoPE achieves state-of-the-art performance for zero-shot DST, beating models that are orders of magnitude larger:

  • MultiWOZ 2.1: MoPE achieved a 57.13% Joint Goal Accuracy (JGA), outperforming both ChatGPT (56.44%) and IC-DST Codex (56.93%), despite MoPE using a backbone model under 10B parameters (ChatGLM-6B).
  • SGD: MoPE achieved a 55.40% JGA, representing a massive 15.6% absolute improvement over the previous state-of-the-art mixture-of-experts model, MoE4DST (39.80%).

Limitations and Future Work

While MoPE delivers impressive results, the authors note a few limitations:

  • Clustering Constraints: The framework currently relies on simple $k$-means clustering. Advanced clustering algorithms like BIRCH, Agglomerative clustering, or Gaussian Mixture Models (GMMs) might partition the slot space more effectively and further boost performance.
  • Backbone Generality: The current implementation was only validated using ChatGLM-6B. Future work includes testing MoPE as a universal plug-in across a wider variety of large language models and diverse NLP tasks.

Summary updated

Abstract

Zero-shot dialogue state tracking (DST) transfers knowledge to unseen domains, reducing the cost of annotating new datasets. Previous zero-shot DST models mainly suffer from domain transferring and partial prediction problems. To address these challenges, we propose Mixture of Prefix Experts (MoPE) to establish connections between similar slots in different domains, which strengthens the model transfer performance in unseen domains. Empirical results demonstrate that MoPE-DST achieves the joint goal accuracy of 57.13% on MultiWOZ2.1 and 55.40% on SGD.
BibTeX
@inproceedings{tang2024mope,
  author = {Tianwen Tang and Tong Zhu and Haodong Liu and Yin Bai and Jia Cheng and Wenliang Chen},
  title = {{MoPE: Mixture of Prefix Experts for Zero-Shot Dialogue State Tracking}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure