Top-N-Rank: A Scalable List-wise Ranking Method for Recommender Systems

Junjie Liang, Jinlong Hu, Shoubin Dong, Vasant G. Honavar

Abstract

In modern recommender systems, the primary objective is to accurately rank the top $N$ items rather than the entire catalog. Traditional list-wise Learning-to-Rank (LTR) methods often struggle with this focus, as they are susceptible to noise from low-ranked items and suffer from quadratic computational complexity. This paper presents Top-N-Rank, a novel family of list-wise ranking models that optimize a truncated version of the Discounted Cumulative Gain (DCG) objective. By utilizing the Rectified Linear Unit (ReLU) as a smoothing function and a sorting-based gradient calculation, Top-N-Rank achieves state-of-the-art ranking performance while reducing complexity to linear $O(m)$. Evaluations on large-scale datasets demonstrate that focusing on the "top-heavy" portion of the list significantly improves recommendation quality.

Introduction

Recommender systems typically present users with a short list of items. Consequently, the accuracy of the top few positions is far more critical than the ranking of items further down the list. However, many existing LTR methods attempt to optimize the entire list, which introduces two main challenges:

  1. Noise Sensitivity: Irrelevant or low-ranked items with unreliable feedback can degrade the model's ability to learn the preferences for top-tier items.
  2. Scalability Bottlenecks: List-wise objectives often require $O(m^2)$ operations per user, where $m$ is the number of items, making them difficult to scale to massive datasets.

Top-N-Rank addresses these issues by truncating the objective function and employing efficient optimization techniques.

Methods: Truncation and ReLU Smoothing

The core of the proposed approach is the wDCG@N (weighted Discounted Cumulative Gain at N) objective function. This function incorporates top-N truncation to ignore items outside the desired recommendation window and allows for weighted feedback to handle diverse interaction types (e.g., clicks vs. purchases).

To optimize this non-smooth objective, the authors introduce Top-N-Rank.ReLU. While the Sigmoid function is a common choice for smoothing in ranking tasks, the authors demonstrate that ReLU is superior. ReLU's one-sided nature naturally filters out the influence of low-ranked items. Furthermore, by sorting items by predicted scores, the authors derived a gradient accumulation method that reduces complexity from $O(m^2)$ to $O(m)$.

Algorithm 1: Learning latent factors with linear complexity.

As shown in Algorithm 1, the update rules for user and item latent factors leverage the sorted order of items to compute gradients efficiently. This linear scalability allows the model to handle industry-scale datasets like MovieLens 20M.

Experimental Setup

The models were evaluated on two benchmark datasets:

  • MovieLens (20M): 20 million ratings from ~138,000 users.
  • Amazon Video Games: Over 1 million ratings from ~7,000 users.

The primary evaluation metric used was Normalized Discounted Cumulative Gain (NDCG) at various cut-offs (1, 3, 5, 10, and 20).

Key Results

Top-N-Rank.ReLU consistently outperformed state-of-the-art baselines, including CLiMF, xCLiMF, and ListRank. The results indicate that the combination of top-N truncation and ReLU smoothing provides a significant advantage in ranking precision.

Comparison of Top-N-Rank.ReLU against state-of-the-art baselines on Amazon Video Games and MovieLens datasets.

The experimental results in the table above highlight that Top-N-Rank.ReLU achieves the highest NDCG scores across all tested thresholds. A key finding is the "Truncation Benefit": models that explicitly ignore the bottom of the list (Top-N variants) consistently outperform those that do not, confirming that focusing on the top of the list is a superior strategy for recommendation.

Conclusion

Top-N-Rank demonstrates that list-wise ranking can be both highly accurate and computationally efficient. By shifting the focus to the top $N$ items and utilizing ReLU for gradient smoothing, the model achieves linear scalability without sacrificing performance. Future work includes extending the model to handle multi-modal features via tensor models and integrating deep neural networks to capture non-linear user-item interactions.

Summary updated

BibTeX
@inproceedings{liang2018top,
  author = {Junjie Liang and Jinlong Hu and Shoubin Dong and Vasant G. Honavar},
  title = {{Top-N-Rank: A Scalable List-wise Ranking Method for Recommender Systems}},
  booktitle = {IEEE BigData},
  year = {2018},
  doi = {10.1109/bigdata.2018.8621994},
}
Figure