Abstract
Goal-oriented dialogues, such as recommendation and negotiation, often require balancing multiple conflicting objectives. Conventional approaches typically train separate policies for each predefined objective trade-off, which is computationally costly and scales poorly. In this work, we pursue a single dialogue policy that can dynamically adapt to varying objective preferences at inference time without retraining. To address challenges in optimization and knowledge utilization, we propose the Preference Adaptive Dialogue Policy Planner (PADPP). This framework introduces a novel optimization scheme leveraging information from previously updated objective weights and utilizes Generalized Policy Improvement (GPI) to ensure effective knowledge transfer. Experimental results demonstrate that PADPP achieves superior adaptability and performance compared to state-of-the-art approaches, offering a scalable solution for multi-objective, goal-oriented dialogues.
Introduction: The Multi-Objective Dilemma
In real-world goal-oriented dialogues, agents rarely pursue a single, isolated goal. A recommendation system must balance user satisfaction with the frequency of promoted items. Similarly, a negotiation agent must navigate the tension between maximizing profit (Price Gain) and maintaining fairness to ensure a deal is actually reached.
Traditionally, Reinforcement Learning (RL) policies are trained for a specific, static trade-off—for instance, 70% profit and 30% fairness. If business requirements change, the entire model must be retrained from scratch. This approach is computationally expensive and fails to handle real-time strategic shifts.
The PADPP Framework
The Preference Adaptive Dialogue Policy Planner (PADPP) addresses these limitations by acting as a "plug-in" planner that guides a larger Language Model (LLM). Instead of a static policy, PADPP learns a single policy that is conditioned on a preference weight vector $w$, provided at the moment of conversation.
Architecture and Optimization
PADPP utilizes a RoBERTa-based encoder and a Deep Q-Network (DQN) to evaluate potential dialogue actions. The framework is divided into two primary phases: training and inference.
- Multi-Objective Reinforcement Learning (MORL): The agent receives a vector of rewards representing different objectives rather than a single scalar.
- Knowledge Reuse Mechanism: To accelerate learning, PADPP maintains a memory buffer of transitions associated with various preference weights. This allows the model to "remember" how it handled different trade-offs in the past.
- Generalized Policy Improvement (GPI): By employing GPI, the authors ensure that the learned policy is at least as effective as any single policy optimized for a specific weight combination. This "distillation" of past experiences allows the model to generalize across the entire preference space efficiently.
Experimental Evaluation
The researchers evaluated PADPP on two benchmark datasets: Craigslist Bargain (a negotiation task) and DuRecDial 2.0 (a recommendation task).
Key Results
- Superior Adaptability: On the Craigslist Bargain dataset, PADPP achieved a Success Rate (SR) of 0.427 under uniform weights, more than doubling the performance of a standard Double DQN (0.194).
- Computational Efficiency: PADPP required only 3.71 hours for total training and evaluation. In contrast, baseline methods like TRIP or PPDPP required 10 to 15 hours because they necessitated separate training runs for every specific objective setting.
- Pareto Front Excellence: Ablation studies confirmed that the knowledge reuse mechanism is vital. PADPP consistently established a superior "solution frontier"—the boundary of optimal trade-offs—compared to versions of the model without knowledge reuse.
- Human Alignment: In side-by-side human evaluations, PADPP was consistently rated higher for "Negotiation Equity" and "Deal Achievement" than other state-of-the-art planners.
Conclusion
PADPP represents a significant step forward in creating flexible, goal-oriented dialogue agents. By moving from static, single-objective policies to a single, preference-adaptive planner, the framework offers a scalable and efficient solution for complex real-world interactions. The ability to shift strategies mid-conversation without retraining opens new possibilities for personalized and context-aware AI assistants.
Summary updated