Enhancing Goal-oriented Proactive Dialogue Systems via Dynamic Multi-dimensional Consistency Optimization

Didi Zhang, Yaxin Fan, Peifeng Li, Qiaoming ZhuEMNLP 2025

TL;DR

This paper introduces DMCRL, a novel framework that dynamically optimizes multi-dimensional consistency in goal-oriented proactive dialogue systems. By adaptively weighting different consistency factors (user profile, history, domain knowledge, subgoal) and employing counterfactual reasoning, DMCRL significantly improves response quality, factual accuracy, and goal achievement.

What Problem Does This Paper Address?

Goal-oriented proactive dialogue systems (GPDS) aim to guide users towards specific objectives through carefully planned conversations. While existing systems focus on planning optimal dialogue paths, they often struggle with maintaining consistency across various contextual elements: the user's profile, the ongoing dialogue history, the system's domain knowledge, and the current subgoal. Inconsistencies in these areas can lead to disjointed, ineffective, and frustrating user experiences. For instance, a system might ignore user preferences, provide factually incorrect information, or fail to progress towards the intended goal. Current approaches often treat all consistency dimensions equally, overlooking the fact that their importance fluctuates dynamically throughout a conversation.

Methods

The proposed solution is the Dynamic Multi-dimensional Consistency Reinforcement Learning (DMCRL) framework, which operates in two main stages:

  1. Supervised Fine-Tuning (SFT): An open-source Large Language Model (LLM) is fine-tuned on a dialogue dataset to establish a baseline for response generation. This stage minimizes the negative log-likelihood of ground-truth responses.
  2. Dynamic Consistency Reinforcement Learning (DCRL): This stage refines the model's ability to generate consistent and well-balanced responses. Key innovations include:
  3. Dynamic Weight Allocation (DWA): A model is trained to dynamically assign importance weights to the four consistency dimensions (user profile, dialogue history, domain knowledge, subgoal) based on the current conversational context. This is achieved by leveraging ChatGPT for initial annotation of these weights, followed by training a DWA model to predict them.
  4. Counterfactual Consistency Reward (Ccf): This mechanism enhances the model's ability to identify and optimize consistency biases. It involves simulating hypothetical scenarios by slightly altering contextual elements (e.g., user preferences, domain knowledge) and evaluating whether the model's response remains consistent and logical under these perturbations. This approach is inspired by causal inference.
  5. Adaptive Consistency Reward: The overall reward function integrates the dynamically weighted consistency scores and the counterfactual consistency reward, guiding the reinforcement learning process.
  6. Proximal Policy Optimization (PPO): This standard reinforcement learning algorithm is used to optimize the model's policy, balancing stability and sample efficiency.

Datasets

The experiments were conducted on two benchmark datasets:

  • DuRecDial (Liu et al., 2020): A Chinese dataset for goal-oriented proactive dialogue systems.
  • DuRecDial 2.0 (Liu et al., 2021): An English dataset for the same task.

The paper notes that data preprocessing and partitioning followed the strategies used in prior work to ensure fair comparisons.

Key Insights and Ideas

  • Dynamic Consistency is Crucial: The most significant insight is that the importance of different consistency dimensions (user profile, history, knowledge, subgoal) is not static but changes dynamically throughout a conversation. Treating them equally can lead to suboptimal or even detrimental results.
  • Counterfactual Reasoning for Robustness: Simulating "what-if" scenarios through counterfactual analysis helps the model learn to maintain consistency even when faced with slightly altered contexts, making it more robust to subtle shifts in the conversation.
  • ChatGPT as a Reliable Annotator: The study demonstrates that ChatGPT can reliably annotate turn-level consistency weights, providing a scalable and effective way to generate training data for dynamic weight allocation.
  • Balancing Consistency and Diversity: While enhancing consistency, the DMCRL framework aims to avoid compromising response diversity, ensuring that generated content remains engaging.

Key Results

  • Significant Performance Gains: DMCRL consistently improved performance across various metrics (Word-level F1, BLEU-2, Knowledge F1, Goal Success Rate) on both DuRecDial and DuRecDial 2.0 datasets, outperforming baselines and static consistency correction methods.
  • LLM Enhancement: Even state-of-the-art LLMs (LLaMA3, Mistral, Phi-3.5, Qwen2.5, DeepSeek) benefited from DMCRL, showing further improvements in consistency and goal achievement.
  • Ablation Studies Confirm Component Importance: Removing any component of DMCRL (User Profile, Dialogue History, Domain Knowledge, Subgoal, Counterfactual Reward, Dynamic Weight Allocation) led to a significant decrease in performance, highlighting the critical contribution of each part.
  • Human Evaluation Supports DMCRL: Human evaluators consistently preferred DMCRL-generated responses over those from baseline and static correction models, citing better alignment with user profiles, dialogue history, domain knowledge, and subgoals.
  • Reduced Subgoal Failures: DMCRL significantly reduced subgoal failure rates, indicating its effectiveness in guiding the dialogue towards the intended objective.

Limitations and Future Work

The authors acknowledge two primary limitations:

  • Reliance on ChatGPT for Annotation: While effective, using ChatGPT for automatic annotation might introduce biases inherent to large language models, potentially overlooking subtle nuances that human annotators might capture.
  • Lack of Explicit Conflict Resolution: The framework dynamically balances consistency dimensions but does not explicitly address potential conflicts between them (e.g., when satisfying user profile information contradicts dialogue history).

Future work could explore conflict resolution mechanisms to further enhance the framework's robustness.

Summary updated

BibTeX
@inproceedings{zhang2025enhancing,
  author = {Didi Zhang and Yaxin Fan and Peifeng Li and Qiaoming Zhu},
  title = {{Enhancing Goal-oriented Proactive Dialogue Systems via Dynamic Multi-dimensional Consistency Optimization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure