In-Context Learning for Few-Shot Dialogue State Tracking

Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, Mari Ostendorf

Abstract

Dialogue State Tracking (DST) is a critical component of task-oriented dialogue systems, traditionally requiring extensive labeled data and fine-tuning. This paper introduces IC-DST, a framework that reformulates DST as a Text-to-SQL task and leverages in-context learning (ICL) with large language models (LLMs) pre-trained on code. By predicting state changes (deltas) rather than full states and using a specialized retriever for few-shot examples, IC-DST achieves state-of-the-art results on the MultiWOZ benchmark in few-shot and zero-shot settings without any parameter updates.

Introduction

Dialogue State Tracking (DST) serves as the "brain" of task-oriented chatbots, maintaining a record of user requirements (e.g., "finding a cheap Italian restaurant") throughout a conversation. Building effective DST modules typically requires massive amounts of hand-annotated data and extensive fine-tuning, making them brittle to new domains or slot additions. The authors propose IC-DST to address these challenges through the In-Context Learning (ICL) paradigm, where a model learns to perform a task simply by observing a few examples in its prompt.

Methodology

The IC-DST framework introduces several key innovations to optimize ICL for structured dialogue tasks:

  1. DST as Text-to-SQL: The task is reformulated as generating a SQL query. Since models like OpenAI’s Codex are pre-trained on vast amounts of code, they are naturally adept at generating structured SQL compared to arbitrary text formats.
  2. Predicting State Changes (Deltas): Instead of predicting the entire state at every turn, which can become prohibitively complex, IC-DST predicts only the changes (e.g., updating a food preference). This focus on the current turn's delta improves robustness.
  3. Efficient Context Representation: To stay within token limits, the dialogue history is represented concisely as: [Previous Dialogue State] + [Current System Turn] + [Current User Turn].
  4. Specialized Dialogue Retriever: For few-shot settings, a fine-tuned SBERT retriever identifies labeled dialogue turns with similar state changes to the current test turn, providing the model with highly relevant context.
Overview of the IC-DST framework, showing the retrieval of labeled examples and the inference process to predict state changes in SQL format.

Results and Key Insights

The researchers evaluated IC-DST on the MultiWOZ 2.1 and 2.4 benchmarks, yielding several significant findings:

  • Few-Shot Dominance: With only 1% of training data (approximately 80 dialogues), IC-DST using Codex achieved a 48.35% Joint Goal Accuracy (JGA) on MultiWOZ 2.4, significantly outperforming fine-tuned baselines like DS2-T5 (~36.7%).
  • Zero-Shot Leap: In zero-shot settings, IC-DST improved the state-of-the-art by 10–30% absolute accuracy across various domains, suggesting that dialogue tracking knowledge is latent in large code-trained LMs.
  • Code-Pretraining Advantage: Models trained on code (e.g., Codex) substantially outperformed general language models (e.g., GPT-Neo), highlighting the importance of structured reasoning capabilities for DST.
Comparison of IC-DST against fine-tuned baselines on MultiWOZ 2.1 and 2.4 across different data regimes (1%, 5%, 10%, 100%).

Discussion and Limitations

While IC-DST offers a flexible and data-efficient alternative to traditional DST, it faces challenges regarding inference costs and API dependencies on closed-source models like Codex. Furthermore, the model's performance is sensitive to the quality of retrieved examples; noisy or incorrectly labeled examples in the prompt can lead to propagated errors. Future work may explore the use of smaller, open-source code-heavy models and the system's robustness to noisy inputs like speech-to-text transcripts.

Summary updated

Abstract

Collecting and annotating task-oriented dialogues is time-consuming and costly; thus, zero and few shot learning could greatly benefit dialogue state tracking (DST). In this work, we propose an in-context learning (ICL) framework for zero-shot and few-shot learning DST, where a large pre-trained language model (LM) takes a test instance and a few exemplars as input, and directly decodes the dialogue state without any parameter updates. To better leverage a tabular domain description in the LM prompt, we reformulate DST into a text-to-SQL problem. We also propose a novel approach to retrieve annotated dialogues as exemplars. Empirical results on MultiWOZ show that our method IC-DST substantially outperforms previous fine-tuned state-of-the-art models in few-shot settings. In addition, we test IC-DST in zero-shot settings, in which the model only takes a fixed task instruction as input, finding that it outperforms previous zero-shot methods by a large margin.
BibTeX
@inproceedings{hu2022context,
  author = {Yushi Hu and Chia-Hsuan Lee and Tianbao Xie and Tao Yu and Noah A. Smith and Mari Ostendorf},
  title = {{In-Context Learning for Few-Shot Dialogue State Tracking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=tNMUvmWSfW},
}
Figure