Enhancing the Preference Extractor in Multi-turn Dialogues: From Annotating Disasters to Accurate Preference Extraction

Cheng Wang, Ziru Liu, Pengcheng Tang, Mingyu Zhang, Quanyu Dai, Yue ZhuarXiv 2025 · DOI · arXiv

Introduction

Extracting user preferences in task-oriented dialogue systems—often referred to as Dialogue State Tracking (DST)—is a critical yet challenging task. As conversations progress, users frequently introduce, modify, or retract constraints. Traditional approaches require models and human annotators to process long, multi-turn transcripts. This leads to two significant issues: the "Annotating Disaster," where human cognitive load results in slow and error-prone labeling, and "preference slot oblivion," where Large Language Models (LLMs) lose track of earlier constraints as the context window expands.

The IterChat Framework

To address these challenges, we introduce IterChat, a framework that reframes multi-turn preference extraction as an iterative, single-turn process. Instead of analyzing an entire dialogue history at once, IterChat decomposes the task into two components:

  1. History Preference: A structured summary of all preferences established prior to the current turn.
  2. Current Dialogue Turn: The most recent exchange between the user and the agent.

By focusing only on the incremental change (the StateGain), the framework significantly reduces the complexity for both humans and machines.

The IterChat pipeline consists of three modules: Preference Schema Generation, Dialogue Sampling, and Annotation.

Methodology and Data Generation

The IterChat pipeline (illustrated above) automates the creation of high-quality datasets. It begins with a Preference Schema Generation Module, where an LLM (e.g., GPT-4) defines domain-specific slots and values. The Dialogue Sampling Module then selects random preference states and prompts the LLM to generate a single-turn dialogue that bridges them. Finally, the Annotation Module produces the "Golden Label" by combining the history state with the new information extracted from the generated turn. This "backward" generation ensures that the dialogue is perfectly aligned with the target labels, avoiding the logical inconsistencies common in multi-turn synthetic data.

Experimental Evaluation

We evaluated IterChat across several benchmarks, including MultiWOZ 2.1 and MultiDOGO, alongside a newly created dataset, Foodie.

Annotation Efficiency and Accuracy

One of the most striking results is the impact on human annotation. By simplifying the task to single-turn updates, we observed a 28.4% reduction in total annotation time.

Comparison of annotation time between IterChat and traditional multi-turn formats across four annotators.

As shown in the figure above, annotators consistently completed their tasks faster using the IterChat format. Furthermore, this reduction in cognitive load translated directly into higher quality data: human Exact Match (EM) accuracy improved from 73.42% in the multi-turn format to 84.37% with IterChat.

Model Performance

The benefits extend to model training. Fine-tuning LLMs on the IterChat format yields substantial performance gains. For instance, a Llama-2-13B model achieved an EM of 81.66% on the Foodie dataset when trained with IterChat, compared to only 66.66% using traditional multi-turn data. This suggests that the iterative format helps models overcome "preference slot oblivion" by providing a clearer, more focused learning signal.

Conclusion

IterChat demonstrates that the key to mastering multi-turn dialogue state tracking lies in decomposition. By shifting from holistic transcript analysis to iterative state updates, we can mitigate the "Annotating Disaster" for humans and improve the reliability of LLMs. While future work is needed to handle highly complex or non-linear conversations, IterChat provides a robust foundation for building more efficient and accurate task-oriented dialogue systems.

Summary updated

Abstract

Identifying user preferences in dialogue systems is a pivotal aspect of providing satisfying services. Current research shows that using large language models (LLMs) to fine-tune a task-specific preference extractor yields excellent results in terms of accuracy and generalization. However, the primary challenge stems from the inherent difficulty in obtaining high-quality labeled multi-turn dialogue data. Accurately tracking user preference transitions across turns not only demands intensive domain expertise and contextual consistency maintenance for annotators (termed \textbf{``Annotating Disaster''}) but also complicates model training due to error propagation in sequential dependency learning. Inspired by the observation that multi-turn preference extraction can be decomposed into iterative executions of one-turn extraction processes. We propose a novel dialogue data generation framework named \textbf{IterChat}. First, we construct a new data format that categorizes the dialogue data into attributed historical preferences and one-turn dialogues. This reduces the probability of annotation errors and improves annotation efficiency. Then, to generate a high-quality and diverse dialogue dataset, we adopt GPT4 to pre-define the preference slots in the target preference extractor task and then randomly sample the subset of the slots and their corresponding schema values to create the dialogue datasets. Experimental results indicate that fine-tuning or only few-shot prompting with the new dialogue format yields superior performance compared to the original multi-turn dialogues. Additionally, the new data format improves annotator efficiency with a win rate of 28.4\% higher than the original multi-turn dialogues.
BibTeX
@misc{wang2025enhancing,
  author = {Cheng Wang and Ziru Liu and Pengcheng Tang and Mingyu Zhang and Quanyu Dai and Yue Zhu},
  title = {{Enhancing the Preference Extractor in Multi-turn Dialogues: From Annotating Disasters to Accurate Preference Extraction}},
  year = {2025},
  eprint = {2508.01739},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.01739},
}
Figure