In-Context Learning for Few-Shot Dialogue State Tracking

Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, Mari Ostendorf

Abstract

Collecting and annotating task-oriented dialogues is time-consuming and costly; thus, zero and few-shot learning could greatly benefit dialogue state tracking (DST). In this work, we propose an in-context learning (ICL) framework for zero-shot and few-shot learning DST, where a large pre-trained language model (LM) takes a test instance and a few exemplars as input and directly decodes the dialogue state without any parameter updates. To better leverage a tabular domain description in the LM prompt, we reformulate DST into a text-to-SQL problem. We also propose a novel approach to retrieve annotated dialogues as exemplars. Empirical results on MultiWOZ show that our method, IC-DST, substantially outperforms previous fine-tuned state-of-the-art models in few-shot settings. In addition, IC-DST outperforms previous zero-shot methods by a large margin.

1. Introduction

Dialogue State Tracking (DST) is a core component of task-oriented dialogue systems, responsible for maintaining a representation of the user's goals and constraints throughout a conversation. For instance, if a user requests a "cheap Italian restaurant," the DST must accurately update the food slot to Italian and the price range slot to cheap within the internal state.

Traditionally, DST models rely on supervised fine-tuning on massive, hand-annotated datasets. However, this approach is brittle; adding new domains or modifying existing schemas often necessitates expensive re-annotation and model retraining. To address these challenges, we explore In-Context Learning (ICL), which allows large language models (LLMs) to perform tasks using only a few demonstrations (exemplars) provided in the prompt, without any gradient-based updates.

2. The IC-DST Framework

We introduce IC-DST, a framework designed to maximize the potential of LLMs for few-shot and zero-shot DST. The framework is built upon three primary pillars:

2.1 DST as Text-to-SQL

A significant challenge in DST is mapping natural language to a structured state. We observe that LLMs like Codex are pre-trained on vast repositories of code, making them highly proficient at generating structured languages. By reformulating the DST task as a text-to-SQL problem—where the dialogue state is treated as a database table and user utterances are translated into SQL queries—we leverage the model's inherent understanding of structured logic. This approach proves more effective than asking the model to output arbitrary text formats.

2.2 State-Based Context Management

Standard DST models often process the entire dialogue history, which can quickly exceed the context window of an LLM. IC-DST employs a more efficient "state-based" context. Instead of the full history, the prompt includes the previous dialogue state and the current turn's utterances. The model is then tasked with predicting the "delta"—the updates required to transition from the previous state to the current one. This approach reduces complexity and focuses the model's attention on the most recent information.

2.3 Dynamic Exemplar Retrieval

The quality of in-context learning is highly dependent on the relevance of the provided exemplars. We utilize a retrieval-based approach using a fine-tuned SBERT model. For a given test instance, the retriever searches a small pool of labeled dialogues to find examples where the dialogue state changes are semantically similar to the current turn. This ensures that the LLM is provided with relevant "how-to" demonstrations for specific user requests.

3. Experimental Evaluation

We evaluated IC-DST on the MultiWOZ 2.1 and 2.4 datasets, which are standard benchmarks for multi-domain DST.

3.1 Few-Shot Performance

In few-shot settings using only 1% of the training data (approximately 80 dialogues), IC-DST using Codex achieved a Joint Goal Accuracy (JGA) of 43.1% on MultiWOZ 2.1. This performance exceeds that of previous state-of-the-art models that were fully fine-tuned on the same amount of data, demonstrating the efficiency of the ICL approach.

3.2 Zero-Shot Performance

In the zero-shot setting, where the model receives only a task instruction and no exemplars, IC-DST achieved a JGA of 35.3% on MultiWOZ 2.4. This represents a significant breakthrough, as previous zero-shot methods typically struggled to reach the 10-20% JGA range.

3.3 Model Scaling

Our results indicate a strong correlation between model size and DST performance. Larger models like Codex significantly outperformed smaller alternatives like GPT-Neo and CodeGen. This highlights that the reasoning required to map messy natural language to rigid state updates scales with model size.

4. Discussion and Limitations

While IC-DST demonstrates remarkable performance without parameter updates, several challenges remain:

  • Latency: Inference via large-scale APIs (e.g., Codex) can take several seconds per turn, which may be prohibitive for real-time voice assistants.
  • Cost: High-volume usage of commercial LLM APIs can be expensive compared to deploying smaller, specialized models.
  • Retriever Sensitivity: The model's accuracy is tied to the retriever's performance; if the retriever pulls a noisy or incorrectly labeled example, the LLM is likely to mimic that error.

5. Conclusion

IC-DST demonstrates that In-Context Learning, combined with text-to-SQL reformulation and intelligent exemplar retrieval, provides a powerful alternative to traditional fine-tuning for Dialogue State Tracking. This approach significantly reduces the data requirements for building robust dialogue systems and sets a new state-of-the-art for few-shot and zero-shot performance in the field.

Summary updated

Abstract

Collecting and annotating task-oriented dialogues is time-consuming and costly; thus, zero and few shot learning could greatly benefit dialogue state tracking (DST). In this work, we propose an in-context learning (ICL) framework for zero-shot and few-shot learning DST, where a large pre-trained language model (LM) takes a test instance and a few exemplars as input, and directly decodes the dialogue state without any parameter updates. To better leverage a tabular domain description in the LM prompt, we reformulate DST into a text-to-SQL problem. We also propose a novel approach to retrieve annotated dialogues as exemplars. Empirical results on MultiWOZ show that our method IC-DST substantially outperforms previous fine-tuned state-of-the-art models in few-shot settings. In addition, we test IC-DST in zero-shot settings, in which the model only takes a fixed task instruction as input, finding that it outperforms previous zero-shot methods by a large margin.
BibTeX
@inproceedings{hu2022context,
  author = {Yushi Hu and Chia-Hsuan Lee and Tianbao Xie and Tao Yu and Noah A. Smith and Mari Ostendorf},
  title = {{In-Context Learning for Few-Shot Dialogue State Tracking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=tNMUvmWSfW},
}
Figure