Listwise Preference Alignment Optimization for Tail Item Recommendation

Zihao Li, Chao Yang, Tong Zhang, Yakun Chen, Xianzhi Wang, Guandong Xu, Daoyi Dong

Abstract

Preference alignment has achieved greater success on Large Language Models (LLMs) and drawn broad interest in recommendation research. Existing preference alignment methods for recommendation either require explicit reward modeling or only support pairwise preference comparison. The former directly increases substantial computational costs, while the latter hinders training efficiency on negative samples. Moreover, no existing effort has explored preference alignment solutions for tail-item recommendation. To bridge the above gaps, we propose LPO4Rec, which extends the Bradley-Terry model from pairwise comparison to listwise comparison, to improve the efficiency of model training. Specifically, we derive a closed form optimal policy to enable more efficient and effective training without explicit reward modeling. We also present an adaptive negative sampling and reweighting strategy to prioritize tail items during optimization and enhance performance in tail-item recommendations. Besides, we theoretically prove that optimizing the listwise preference optimization (LPO) loss is equivalent to maximizing the upper bound of the optimal reward. Our experiments on three public datasets show that our method outperforms 10 baselines by a large margin, achieving up to 50% performance improvement while reducing 17.9% GPU memory usage when compared with direct preference optimization (DPO) in tail-item recommendation.

Introduction

In recommendation systems, the "long tail" phenomenon presents a significant challenge. Popular "head items" dominate interaction data, causing models to prioritize them at the expense of "tail items"—the vast majority of the catalog. This imbalance limits the diversity and fairness of recommendations. While preference alignment techniques like Direct Preference Optimization (DPO) have shown promise in aligning model outputs with human preferences in LLMs, their application to recommendation systems remains limited. Existing methods often suffer from high computational overhead due to explicit reward modeling or training inefficiencies from pairwise comparisons. Furthermore, the specific needs of tail-item recommendation have been largely overlooked in the context of preference alignment.

Methodology: LPO4Rec

To address these limitations, we introduce LPO4Rec (Listwise Preference Optimization for Recommendation). The framework is built on three main pillars:

1. Listwise Preference Alignment

Unlike traditional pairwise alignment (e.g., DPO) which compares a single positive item against a single negative item, LPO4Rec employs a listwise approach. By extending the Bradley-Terry model, it compares a positive item against a set of $K$ negative items simultaneously. We derived a closed-form optimal policy that allows the model to be trained directly via maximum likelihood, eliminating the need for a separate, resource-intensive reward model.

Comparison of Pair-wise Direct Preference Optimization (DPO) and the proposed List-wise Preference Optimization (LPO) alignment strategies.

2. Adaptive Negative Sampling

To improve the model's discriminative power, we utilize an adaptive negative sampling strategy. Using the Gumbel-Softmax trick, the model dynamically selects "hard" negative samples from the head-item set. This forces the model to learn subtle distinctions between popular items and the items a user truly prefers.

3. Adaptive Loss Reweighting

To specifically target tail-item performance, LPO4Rec incorporates an adaptive loss reweighting mechanism. During training, higher weights are assigned to loss terms involving tail items. This ensures that gradient updates are biased toward improving the representation and recommendation of rare items.

Experimental Results

We evaluated LPO4Rec on three Amazon datasets: Beauty, Toys, and Sports. Items were categorized into "head" and "tail" based on the 80/20 Pareto principle.

Performance Gains

LPO4Rec consistently outperformed ten baseline models, including specialized tail-item recommendation methods and LLM-based alignment strategies. Notably, on the Amazon Sports dataset, LPO4Rec achieved a 50% improvement in Recall@20 for tail items compared to the strongest baseline. Visual analysis of probability distributions confirmed that LPO4Rec effectively shifts recommendation probabilities toward tail items.

Computational Efficiency

A key advantage of LPO4Rec is its efficiency. By removing the reference model requirement typical of DPO, it maintains a low memory footprint similar to vanilla recommendation models like SASRec.

Comparison of training time (seconds) and GPU memory usage (MiB) across various preference alignment methods.

As shown in the figure above, LPO4Rec reduces GPU memory usage by 17.9% compared to DPO. Furthermore, it is significantly faster than S-DPO (a multi-negative variant of DPO), achieving a 54.2% reduction in training time per epoch.

Conclusion

LPO4Rec provides a robust and efficient framework for aligning recommendation models with user preferences while specifically addressing the long-tail problem. By moving to listwise optimization and employing adaptive strategies, it achieves state-of-the-art performance for tail items without the computational burden of traditional preference alignment methods. Future work will explore the integration of this framework into more complex architectures and its application to cold-start scenarios.

Summary updated

Abstract

Preference alignment has achieved greater success on Large Language Models (LLMs) and drawn broad interest in recommendation research. Existing preference alignment methods for recommendation either require explicit reward modeling or only support pairwise preference comparison. The former directly increases substantial computational costs, while the latter hinders training efficiency on negative samples. Moreover, no existing effort has explored preference alignment solutions for tail-item recommendation. To bridge the above gaps, we propose LPO4Rec, which extends the Bradley-Terry model from pairwise comparison to listwise comparison, to improve the efficiency of model training. Specifically, we derive a closed form optimal policy to enable more efficient and effective training without explicit reward modeling. We also present an adaptive negative sampling and reweighting strategy to prioritize tail items during optimization and enhance performance in tail-item recommendations. Besides, we theoretically prove that optimizing the listwise preference optimization (LPO) loss is equivalent to maximizing the upper bound of the optimal reward. Our experiments on three public datasets show that our method outperforms 10 baselines by a large margin, achieving up to 50% performance improvement while reducing 17.9% GPU memory usage when compared with direct preference optimization (DPO) in tail-item recommendation. Our code is available at https://github.com/Yuhanleeee/LPO4Rec.
BibTeX
@article{li2025listwise,
  author = {Zihao Li and Chao Yang and Tong Zhang and Yakun Chen and Xianzhi Wang and Guandong Xu and Daoyi Dong},
  title = {{Listwise Preference Alignment Optimization for Tail Item Recommendation}},
  journal = {IEEE Transactions on Computational Social Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=u0CQlV5NEV},
}
Figure