Introduction
Extracting user preferences in task-oriented dialogue systems—often referred to as Dialogue State Tracking (DST)—is a critical yet challenging task. As conversations progress, users frequently introduce, modify, or retract constraints. Traditional approaches require models and human annotators to process long, multi-turn transcripts. This leads to two significant issues: the "Annotating Disaster," where human cognitive load results in slow and error-prone labeling, and "preference slot oblivion," where Large Language Models (LLMs) lose track of earlier constraints as the context window expands.
The IterChat Framework
To address these challenges, we introduce IterChat, a framework that reframes multi-turn preference extraction as an iterative, single-turn process. Instead of analyzing an entire dialogue history at once, IterChat decomposes the task into two components:
- History Preference: A structured summary of all preferences established prior to the current turn.
- Current Dialogue Turn: The most recent exchange between the user and the agent.
By focusing only on the incremental change (the StateGain), the framework significantly reduces the complexity for both humans and machines.
Methodology and Data Generation
The IterChat pipeline (illustrated above) automates the creation of high-quality datasets. It begins with a Preference Schema Generation Module, where an LLM (e.g., GPT-4) defines domain-specific slots and values. The Dialogue Sampling Module then selects random preference states and prompts the LLM to generate a single-turn dialogue that bridges them. Finally, the Annotation Module produces the "Golden Label" by combining the history state with the new information extracted from the generated turn. This "backward" generation ensures that the dialogue is perfectly aligned with the target labels, avoiding the logical inconsistencies common in multi-turn synthetic data.
Experimental Evaluation
We evaluated IterChat across several benchmarks, including MultiWOZ 2.1 and MultiDOGO, alongside a newly created dataset, Foodie.
Annotation Efficiency and Accuracy
One of the most striking results is the impact on human annotation. By simplifying the task to single-turn updates, we observed a 28.4% reduction in total annotation time.
As shown in the figure above, annotators consistently completed their tasks faster using the IterChat format. Furthermore, this reduction in cognitive load translated directly into higher quality data: human Exact Match (EM) accuracy improved from 73.42% in the multi-turn format to 84.37% with IterChat.
Model Performance
The benefits extend to model training. Fine-tuning LLMs on the IterChat format yields substantial performance gains. For instance, a Llama-2-13B model achieved an EM of 81.66% on the Foodie dataset when trained with IterChat, compared to only 66.66% using traditional multi-turn data. This suggests that the iterative format helps models overcome "preference slot oblivion" by providing a clearer, more focused learning signal.
Conclusion
IterChat demonstrates that the key to mastering multi-turn dialogue state tracking lies in decomposition. By shifting from holistic transcript analysis to iterative state updates, we can mitigate the "Annotating Disaster" for humans and improve the reliability of LLMs. While future work is needed to handle highly complex or non-linear conversations, IterChat provides a robust foundation for building more efficient and accurate task-oriented dialogue systems.
Summary updated