Leveraging Unpaired Feedback for Long-Term LLM-based Recommendation Tuning

Jizhi Zhang, Chongming Gao, Wentao Shi, Xin Chen, Jingang Wang, Xunliang Cai, Fuli FengEMNLP 2025

TL;DR

This paper introduces ULRec, a novel framework that fine-tunes Large Language Models (LLMs) for recommendation systems by effectively leveraging unpaired user feedback. By modeling both positive engagement and negative disengagement signals without explicit labels, ULRec significantly improves long-term user satisfaction and breaks echo chambers, outperforming existing methods.

What Problem Does This Paper Address?

Traditional recommendation systems often optimize for short-term metrics like click-through rates, inadvertently leading to "echo chambers" where users are repeatedly shown similar content. This can degrade user satisfaction and limit content diversity over time. While LLMs show promise in recommendation, most existing LLM-based approaches still focus on short-term accuracy, neglecting the crucial aspect of long-term user retention. A key challenge in addressing this is learning from unpaired feedback – implicit signals like continued engagement (positive) or silent disengagement (negative) that lack explicit contrastive labels, making them difficult to use in standard training paradigms.

Methods

The core of ULRec lies in its ability to learn from unpaired feedback. The framework employs a two-stage fine-tuning process for LLMs:

  1. Learning from Positive Feedback: This stage focuses on successful user interactions. The LLM is fine-tuned using step-level data extracted from interaction trajectories. The goal is to teach the LLM to identify items that lead to continued engagement, thus helping to break echo chambers.
  2. Learning from Negative Feedback: This stage tackles the more challenging unpaired negative signals. ULRec leverages the KTO (KTO: Model Alignment as Prospect Theoretic Optimization) algorithm, which is adapted to handle implicit negative feedback without requiring explicit paired comparisons. KTO helps the LLM suppress recommendations that might lead to echo chambers or user disengagement. To ensure training stability and prevent output format collapse, a supervised fine-tuning (SFT) loss is incorporated alongside the KTO loss.

The framework also includes a grounding module to ensure that LLM-generated recommendations are mapped to actual items within the candidate set.

Datasets

The authors used two datasets to build simulated interactive environments for training and evaluation:

  • Steam: A dataset from a gaming scenario.
  • Amazon Book: A book recommendation dataset.

These datasets were used to create simulated environments that mimic real-world user interactions, including signals for reward and item similarity. The training data for the LLM was generated by having a large LLM (LLaMA-3-70B) interact with simulated users using the ReAct framework.

Key Insights and Ideas

  • Unpaired Feedback is Key: The paper strongly argues that effectively modeling unpaired user feedback (both positive and negative) is crucial for improving long-term recommendation performance, especially in dynamic user environments.
  • KTO for Implicit Negatives: The adaptation of the KTO algorithm to handle implicit negative feedback without explicit labels is a significant contribution, enabling LLMs to learn from subtle signals of user dissatisfaction.
  • SFT for Stability: The necessity of combining KTO with an SFT loss is highlighted to prevent output format collapse and ensure stable training, demonstrating a practical consideration for applying these advanced techniques.
  • LLMs Generalize Well: The experiments show that ULRec, even with smaller backbone LLMs, can achieve comparable or better performance than much larger models, underscoring the effectiveness of the feedback-driven fine-tuning approach.

Key Results

  • ULRec consistently outperformed traditional recommendation methods and existing LLM-based approaches (like BiLLP) on both "Len" (interaction length) and "Traj Reward" (cumulative reward) metrics across the Steam and Amazon Book datasets.
  • For example, on the Steam dataset, ULRec achieved a "Len" of 18.09 and "Traj Reward" of 83.66, significantly higher than the "Base" LLM (2.38 and 10.39 respectively).
  • Ablation studies showed that both positive feedback (ULRec-P) and negative feedback (ULRec) contribute significantly to performance improvements.
  • The framework demonstrated robustness across different LLM backbones (e.g., Qwen2.5-7B), indicating broad applicability.

Limitations and Future Work

  • Simulation vs. Real-World: The primary limitation is the reliance on simulated environments due to the high cost of real-world online experiments. The authors suggest that future work should validate ULRec's effectiveness in real-world LLM-based recommendation systems.
  • Model Size: Experiments were mainly conducted on LLMs around 8B parameters. Further research could explore the effectiveness of ULRec with models of varying sizes.
  • Hyperparameter Sensitivity: The paper notes that hyperparameters like β (for KTO) and α (for SFT weight) can be sensitive, suggesting that careful tuning is required.
  • Discrepancy between Simulation and Reality: While the simulation environment was optimized, a gap between simulated and real-world scenarios may still exist, which warrants further investigation.

Summary updated

BibTeX
@inproceedings{zhang2025leveraging,
  author = {Jizhi Zhang and Chongming Gao and Wentao Shi and Xin Chen and Jingang Wang and Xunliang Cai and Fuli Feng},
  title = {{Leveraging Unpaired Feedback for Long-Term LLM-based Recommendation Tuning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure