Experience as Source for Anticipation and Planning: Experiential Policy Learning for Target-driven Recommendation Dialogues

Huy Dao, Yang Deng, Khanh-Huyen Bui, Dung D. Le, Lizi Liao

Introduction: The Challenge of Proactive Recommendation

Traditional Conversational Recommender Systems (CRSs) typically operate reactively, responding to user-expressed preferences. However, many real-world applications require a target-driven approach, where the system proactively steers the conversation toward a specific item or goal. This requires the system to anticipate future user reactions and plan its dialogue strategy accordingly.

Figure 1: An example of a target-driven recommendation dialogue where the system must choose strategies to steer the user toward "KungFu Panda 3".

Current state-of-the-art methods often rely on Large Language Models (LLMs) to perform "rollouts"—simulating entire future conversation trajectories to evaluate potential actions. While effective, this approach suffers from high computational costs, potential hallucinations in simulated user behavior, and a failure to leverage the wealth of existing interaction data.

Methodology: Experiential Policy Learning (EPL)

To address these limitations, researchers have introduced Experiential Policy Learning (EPL). The core philosophy shifts from simulation to retrieval: instead of imagining a hypothetical future, the system "remembers" similar past interactions to guide its current decisions.

T-EPL: Tree-structured Experiential Policy Learning

The practical implementation, T-EPL, integrates this experiential approach into a planning framework. The system architecture consists of four primary components:

  1. Tree Search: Utilizing Monte-Carlo Tree Search (MCTS) to explore potential conversational branches.
  2. Dialogue State Retrieval: A dense retrieval model identifies the $k$ most similar dialogue states from a long-term memory of past experiences.
  3. Assessment: An LLM evaluates the success and quality of these retrieved experiences.
  4. Estimation: The system aggregates these assessments to calculate a "state value," which informs the selection of the next dialogue action.
Figure 2: The T-EPL framework architecture, showing the integration of Tree Search, Dialogue State Retrieval, LLM Assessment, and State Value Estimation.

By replacing multi-turn LLM simulations with efficient retrieval, T-EPL reduces computational complexity from $O(TKH^2)$ to $O(TKH)$, where $H$ is the conversation horizon. This makes the system significantly more viable for real-time applications.

Evaluation and Results

The T-EPL model was evaluated on two major benchmarks: DuRecDial 2.0 (multi-domain) and INSPIRED (sociable movie recommendations).

Performance and Efficiency

T-EPL demonstrated superior performance over existing LLM-based planners. On the DuRecDial 2.0 dataset, it achieved a Subjective Success Rate of 0.833, surpassing the previous state-of-the-art GDP-Zero (0.825).

Crucially, the efficiency gains were substantial. In inference tests on the INSPIRED dataset, T-EPL completed tasks in approximately 84 seconds, compared to 148 seconds for GDP-Zero and 232 seconds for standard MCTS with rollouts. Human evaluations further confirmed that T-EPL produces more satisfying justifications and more coherent topical transitions.

Key Insights

  • Grounded Foresight: Real-world past interactions provide a more reliable basis for anticipation than LLM-generated simulations, which can be prone to unrealistic user modeling.
  • Subjective Success Focus: The framework prioritizes whether a user actually accepts a recommendation (subjective success) rather than just whether the item was mentioned (objective success).
  • Strategic Adaptability: T-EPL can dynamically switch between direct recommendation and "nudging" strategies (e.g., building rapport or providing justifications) based on the context of retrieved successful experiences.

Limitations and Future Directions

While T-EPL offers a significant advancement, it remains dependent on the quality and breadth of its memory. Performance may decline in "cold start" scenarios or when encountering entirely new domains. Future research will likely focus on dynamic memory management, including real-time updates and pruning of the experience base to ensure the system remains relevant and efficient as it scales.

Summary updated

Abstract

Target-driven recommendation dialogues present unique challenges in dialogue management due to the necessity of anticipating user interactions for successful conversations. Current methods face significant limitations: (I) inadequate capabilities for conversation anticipation, (II) computational inefficiencies due to costly simulations, and (III) neglect of valuable past dialogue experiences. To address these limitations, we propose a new framework, Experiential Policy Learning (EPL), for enhancing such dialogues. EPL embodies the principle of Learning From Experience, facilitating anticipation with an experiential scoring function that estimates dialogue state potential using similar past interactions stored in long-term memory. To demonstrate its flexibility, we introduce Tree-structured EPL (T-EPL) as one possible training-free realization with Large Language Models (LLMs) and Monte-Carlo Tree Search (MCTS). T-EPL assesses past dialogue states with LLMs while utilizing MCTS to achieve hierarchical and multi-level reasoning. Extensive experiments on two published datasets demonstrate the superiority and efficacy of T-EPL.
BibTeX
@inproceedings{dao2024experience,
  author = {Huy Dao and Yang Deng and Khanh-Huyen Bui and Dung D. Le and Lizi Liao},
  title = {{Experience as Source for Anticipation and Planning: Experiential Policy Learning for Target-driven Recommendation Dialogues}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.829},
}
Figure