Abstract
Dialogue State Tracking (DST) is a critical component of task-oriented dialogue systems, traditionally requiring extensive labeled data and fine-tuning. This paper introduces IC-DST, a framework that reformulates DST as a Text-to-SQL task and leverages in-context learning (ICL) with large language models (LLMs) pre-trained on code. By predicting state changes (deltas) rather than full states and using a specialized retriever for few-shot examples, IC-DST achieves state-of-the-art results on the MultiWOZ benchmark in few-shot and zero-shot settings without any parameter updates.
Introduction
Dialogue State Tracking (DST) serves as the "brain" of task-oriented chatbots, maintaining a record of user requirements (e.g., "finding a cheap Italian restaurant") throughout a conversation. Building effective DST modules typically requires massive amounts of hand-annotated data and extensive fine-tuning, making them brittle to new domains or slot additions. The authors propose IC-DST to address these challenges through the In-Context Learning (ICL) paradigm, where a model learns to perform a task simply by observing a few examples in its prompt.
Methodology
The IC-DST framework introduces several key innovations to optimize ICL for structured dialogue tasks:
- DST as Text-to-SQL: The task is reformulated as generating a SQL query. Since models like OpenAI’s Codex are pre-trained on vast amounts of code, they are naturally adept at generating structured SQL compared to arbitrary text formats.
- Predicting State Changes (Deltas): Instead of predicting the entire state at every turn, which can become prohibitively complex, IC-DST predicts only the changes (e.g., updating a food preference). This focus on the current turn's delta improves robustness.
- Efficient Context Representation: To stay within token limits, the dialogue history is represented concisely as:
[Previous Dialogue State] + [Current System Turn] + [Current User Turn]. - Specialized Dialogue Retriever: For few-shot settings, a fine-tuned SBERT retriever identifies labeled dialogue turns with similar state changes to the current test turn, providing the model with highly relevant context.
Results and Key Insights
The researchers evaluated IC-DST on the MultiWOZ 2.1 and 2.4 benchmarks, yielding several significant findings:
- Few-Shot Dominance: With only 1% of training data (approximately 80 dialogues), IC-DST using Codex achieved a 48.35% Joint Goal Accuracy (JGA) on MultiWOZ 2.4, significantly outperforming fine-tuned baselines like DS2-T5 (~36.7%).
- Zero-Shot Leap: In zero-shot settings, IC-DST improved the state-of-the-art by 10–30% absolute accuracy across various domains, suggesting that dialogue tracking knowledge is latent in large code-trained LMs.
- Code-Pretraining Advantage: Models trained on code (e.g., Codex) substantially outperformed general language models (e.g., GPT-Neo), highlighting the importance of structured reasoning capabilities for DST.
Discussion and Limitations
While IC-DST offers a flexible and data-efficient alternative to traditional DST, it faces challenges regarding inference costs and API dependencies on closed-source models like Codex. Furthermore, the model's performance is sensitive to the quality of retrieved examples; noisy or incorrectly labeled examples in the prompt can lead to propagated errors. Future work may explore the use of smaller, open-source code-heavy models and the system's robustness to noisy inputs like speech-to-text transcripts.
Summary updated