Tunable LLM-based Proactive Recommendation Agent

Mingze Wang, Chongming Gao, Wenjie Wang, Yangyang Li, Fuli FengACL 2025 · OpenReview

Abstract

Recommender systems are indispensable on various digital platforms. However, traditional methods often reinforce existing user interests, which leads to echo chambers and limits diversity. Proactive Recommendation Systems (PRS) aim to address this issue by cultivating users’ latent interests through multi-step recommendations. Despite advancements, challenges persist particularly in optimizing long-term rewards and adapting to real-time user feedback. In this study, we propose an LLM-based Actor-Critic Agent framework to enhance PRS. This framework utilizes the LLM-based agent to adjust recommendations in real time based on feedback and employs agent-tuning methods to optimize long-term rewards using three proposed reward functions. Extensive experiments validate the significant superiority of this framework over existing methods by optimizing long-term rewards and dynamically evolving with user feedback.

Introduction: The Proactive Recommendation Challenge

Traditional recommender systems are primarily "reactive"—they analyze historical preferences to provide more of the same. While this maximizes short-term satisfaction, it frequently results in echo chambers and the filter bubble effect, where a user's content exposure continually narrows.

Proactive Recommendation Systems (PRS) attempt to break this cycle by "path-planning" a user's interest toward a specific target item. However, current PRS methods face two major hurdles:

  1. Short-sightedness: They often prioritize immediate clicks over long-term strategic goals.
  2. Inflexibility: They struggle to adapt to real-time feedback, failing to pivot when a user rejects a "bridge" recommendation.
The trade-off between user satisfaction and interest exploration.

T-PRA seeks the "Pareto Optimality" where interest is expanded without sacrificing user satisfaction, avoiding both the stagnation of echo chambers and the frustration of irrelevant exploration.

The T-PRA Framework

The authors propose T-PRA, a multi-agent LLM framework that treats proactive recommendation as a sequential decision-making problem.

Overview of the T-PRA framework, including the Actor-Advisor architecture and the Critic-guided tuning process.

1. Actor-Advisor Architecture

Inspired by the "fast and slow thinking" paradigm, the system bifurcates the recommendation process:

  • The Actor (Fast): Generates the immediate next recommendation.
  • The Advisor (Slow): Analyzes real-time feedback and historical data to reason about user interest dynamics. It produces a "thought"—a strategic guidance signal—to steer the Actor.

2. Critic-Guided Optimization

To move beyond static prompt engineering, T-PRA introduces an LLM-based Critic. The Critic evaluates the "advantage value" of specific recommendations, estimating how much an action contributes to the long-term goal compared to the baseline.

3. Agent Tuning with DPO

The framework utilizes Direct Preference Optimization (DPO) to fine-tune both the Actor and Advisor. By comparing "preferred" actions (high advantage value) against "dispreferred" ones as identified by the Critic, the LLMs learn strategic nuances not captured by standard pre-training.

4. Multi-Objective Rewards

Training is driven by three distinct reward functions:

  • Coherence ($R_C$): Ensures smooth transitions between items.
  • Distance to Target ($R_T$): Measures progress toward the goal item.
  • User Acceptance ($R_A$): Predicts user engagement with intermediate recommendations.

Experimental Evaluation

The framework was evaluated on two large-scale datasets: Steam (18,523 interaction sequences) and Amazon-Book (3,374 interaction sequences).

Key Results

  • Significant Performance Gains: T-PRA demonstrated an average 38% increase in effectiveness (measured by Increment of Interest and Increment of Rank) compared to previous state-of-the-art methods.
  • Outperforming Larger Models: Even when using a smaller base model (Llama-3.1-8B), the tuned T-PRA outperformed LLM-IPP (which uses GPT-3.5-turbo) across almost all metrics.
  • Robustness: T-PRA maintained high performance levels even as user history grew longer and more complex, showing superior handling of "noisy" historical data.

Conclusion

T-PRA demonstrates that LLMs can be explicitly tuned for recommendation strategy, transforming them from passive filters into proactive planners. While the agent-tuning process is computationally intensive, the gains in long-term user interest cultivation are substantial. Future research will explore Inverse Reinforcement Learning to better infer user reward functions and apply this framework to broader personalized generation tasks.

Summary updated

Abstract

Recommender systems are indispensable on various digital platforms. However, traditional methods often reinforce existing user interests, which leads to echo chambers and limits diversity. Proactive Recommendation Systems (PRS) aim to address this issue by cultivating users’ latent interests through multi-step recommendations. Despite advancements, challenges persist particularly in optimizing long-term rewards and adapting to real-time user feedback. In this study, we propose an LLM-based Actor-Critic Agent framework to enhance PRS. This framework utilizes the LLM-based agent to adjust recommendations in real time based on feedback and employs agent-tuning methods to optimize long-term rewards using three proposed reward functions. Extensive experiments validate the significant superiority of this framework over existing methods by optimizing long-term rewards and dynamically evolving with user feedback.
BibTeX
@inproceedings{wang2025tunable,
  author = {Mingze Wang and Chongming Gao and Wenjie Wang and Yangyang Li and Fuli Feng},
  title = {{Tunable LLM-based Proactive Recommendation Agent}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=7UnwfCEtA2},
}
Figure