One Planner To Guide Them All ! Learning Adaptive Conversational Planners for Goal-oriented Dialogues

Huy Quang Dao, Lizi Liao2025 · DOI

Abstract

Goal-oriented dialogues, such as recommendation and negotiation, often require balancing multiple conflicting objectives. Conventional approaches typically train separate policies for each predefined objective trade-off, which is computationally costly and scales poorly. In this work, we pursue a single dialogue policy that can dynamically adapt to varying objective preferences at inference time without retraining. To address challenges in optimization and knowledge utilization, we propose the Preference Adaptive Dialogue Policy Planner (PADPP). This framework introduces a novel optimization scheme leveraging information from previously updated objective weights and utilizes Generalized Policy Improvement (GPI) to ensure effective knowledge transfer. Experimental results demonstrate that PADPP achieves superior adaptability and performance compared to state-of-the-art approaches, offering a scalable solution for multi-objective, goal-oriented dialogues.

Introduction: The Multi-Objective Dilemma

In real-world goal-oriented dialogues, agents rarely pursue a single, isolated goal. A recommendation system must balance user satisfaction with the frequency of promoted items. Similarly, a negotiation agent must navigate the tension between maximizing profit (Price Gain) and maintaining fairness to ensure a deal is actually reached.

Figure 1: An example of a negotiation dialogue where the agent must balance profit and fairness. Shifting priorities at inference time allows the agent to reach a deal by accepting a fair price instead of sticking to a rigid, high-profit target.

Traditionally, Reinforcement Learning (RL) policies are trained for a specific, static trade-off—for instance, 70% profit and 30% fairness. If business requirements change, the entire model must be retrained from scratch. This approach is computationally expensive and fails to handle real-time strategic shifts.

The PADPP Framework

The Preference Adaptive Dialogue Policy Planner (PADPP) addresses these limitations by acting as a "plug-in" planner that guides a larger Language Model (LLM). Instead of a static policy, PADPP learns a single policy that is conditioned on a preference weight vector $w$, provided at the moment of conversation.

Architecture and Optimization

PADPP utilizes a RoBERTa-based encoder and a Deep Q-Network (DQN) to evaluate potential dialogue actions. The framework is divided into two primary phases: training and inference.

Figure 2: The PADPP architecture. During training, the model uses a buffer and Generalized Policy Improvement (GPI) to reuse knowledge from past preference settings. At inference time, the LM Planner can switch between different action strategies simply by changing the input preference weights.
  1. Multi-Objective Reinforcement Learning (MORL): The agent receives a vector of rewards representing different objectives rather than a single scalar.
  2. Knowledge Reuse Mechanism: To accelerate learning, PADPP maintains a memory buffer of transitions associated with various preference weights. This allows the model to "remember" how it handled different trade-offs in the past.
  3. Generalized Policy Improvement (GPI): By employing GPI, the authors ensure that the learned policy is at least as effective as any single policy optimized for a specific weight combination. This "distillation" of past experiences allows the model to generalize across the entire preference space efficiently.

Experimental Evaluation

The researchers evaluated PADPP on two benchmark datasets: Craigslist Bargain (a negotiation task) and DuRecDial 2.0 (a recommendation task).

Key Results

  • Superior Adaptability: On the Craigslist Bargain dataset, PADPP achieved a Success Rate (SR) of 0.427 under uniform weights, more than doubling the performance of a standard Double DQN (0.194).
  • Computational Efficiency: PADPP required only 3.71 hours for total training and evaluation. In contrast, baseline methods like TRIP or PPDPP required 10 to 15 hours because they necessitated separate training runs for every specific objective setting.
  • Pareto Front Excellence: Ablation studies confirmed that the knowledge reuse mechanism is vital. PADPP consistently established a superior "solution frontier"—the boundary of optimal trade-offs—compared to versions of the model without knowledge reuse.
  • Human Alignment: In side-by-side human evaluations, PADPP was consistently rated higher for "Negotiation Equity" and "Deal Achievement" than other state-of-the-art planners.

Conclusion

PADPP represents a significant step forward in creating flexible, goal-oriented dialogue agents. By moving from static, single-objective policies to a single, preference-adaptive planner, the framework offers a scalable and efficient solution for complex real-world interactions. The ability to shift strategies mid-conversation without retraining opens new possibilities for personalized and context-aware AI assistants.

Summary updated

Abstract

Goal-oriented dialogues, such as recommendation and negotiation, often require balancing multiple conflicting objectives.Conventional approaches typically train separate policies for each predefined objective trade-off, which is computationally costly and scales poorly.In this work, we pursue a single dialogue policy that can dynamically adapt to varying objective preferences at inference time without retraining.This raises several challenges in terms of both (1) optimization strategy and (2) knowledge utilization.To address these, we propose a novel policy learning framework, Preference Adaptive Dialogue Policy Planner (PADPP), for multi-objective goal-oriented dialogues.Specifically, to tackle the former, we introduce a novel optimization scheme, which leverages information gained from training the model on previously updated objective weights, accelerating the learning capability on new weight settings.To address the latter, we utilize Generalized Policy Improvement (GPI) to ensure the effectiveness of leveraged knowledge.Experimental results demonstrate that PADPP achieves superior adaptability and performance compared to state-of-the-art approaches, offering a scalable and flexible solution for multiobjective, goal-oriented dialogues 1 . InferenceRoBERTa DQN LM Planner Action 1 (, , ) Action 2 (, , ) Recommendation Dialogue: DuRecDial 2.0
BibTeX
@misc{dao2025one,
  author = {Huy Quang Dao and Lizi Liao},
  title = {{One Planner To Guide Them All ! Learning Adaptive Conversational Planners for Goal-oriented Dialogues}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1123},
}
Figure