RaCT: Toward Amortized Ranking-Critical Training For Collaborative Filtering

Sam Lobel, Chunyuan Li, Jianfeng Gao, Lawrence CarinICLR 2020 · OpenReview

TL;DR

RaCT (Ranking-Critical Training) introduces an actor-critic framework for collaborative filtering that allows models to optimize directly for non-differentiable ranking metrics like NDCG. By training a "critic" network to approximate these metrics using a clever set of feature-based statistics, the authors significantly boost the performance of standard Variational Autoencoders (VAEs) and other latent variable models across massive real-world datasets.

The Problem: Likelihood vs. Ranking

In the world of recommender systems, we don't just want to predict if a user will like an item; we want the items they like most to appear at the very top of the list. This is fundamentally a ranking problem.

However, most modern deep learning models for recommendation (like VAEs) are trained using Maximum Likelihood Estimation (MLE). They try to maximize the probability of observed user-item interactions. The "gap" is that a model can get very good at predicting likelihood while still being mediocre at ranking.

Likelihood vs Ranking

As illustrated above, likelihood can be a liar. A model with a worse negative log-likelihood (NLL) can actually produce a better recommendation list (perfect NDCG). This proves that MLE-based training has a fundamental ceiling.

Why don't we just optimize for ranking metrics like Normalized Discounted Cumulative Gain (NDCG) directly? Because ranking involves sorting, and sorting is non-differentiable. You can't take a gradient through a "sort" operation, which leaves standard backpropagation dead in its tracks. Previous "Learning-to-Rank" (L2R) methods tried to solve this with relaxations or pairwise comparisons, but these often struggle to scale to the millions of items found in modern datasets.

Methods: The Actor-Critic Paradigm

To bridge this gap, the authors propose RaCT, which borrows the Actor-Critic paradigm from Reinforcement Learning.

Actor-Critic Architecture
  1. The Actor: This is your actual recommendation model (e.g., a VAE). It takes a user's history and predicts a probability distribution over all items.
  2. The Critic: This is a separate neural network designed to act as a differentiable "proxy" for the ranking metric. Instead of the non-differentiable NDCG "Oracle," the Actor talks to the Critic.

The "Secret Sauce": Feature-based Critic

A naive critic would try to look at the raw prediction vector (which could be 40,000+ dimensions) and the ground truth to guess the NDCG. This is computationally impossible and prone to overfitting. Instead, the authors use a feature-based critic that looks at three "sufficient statistics":

  • $L_E$ (Negative Log-Likelihood): How well the model is currently predicting the items.
  • $|H_0|$ and $|H_1|$: The number of unobserved and observed items for that specific user.

These features provide context. For example, ranking is "easier" when a user has many liked items to find ($|H_0|$ is large) and "harder" when we have very little history to go on ($|H_1|$ is small). By including $|H_0|$ and $|H_1|$ in the Critic's input, the model learns that an NLL of 0.5 means something very different for a "power user" versus a "casual user."

The Training Pipeline

Training happens in three stages:

  1. Actor Pre-training: Train the VAE normally using MLE.
  2. Critic Pre-training: Fix the Actor, and train the Critic to predict the true NDCG scores of the Actor's outputs.
  3. Alternative Training: Fine-tune both. The Actor tries to change its predictions to make the Critic give it a higher score, while the Critic updates itself to remain an accurate predictor of the true ranking metric.

Key Results and Insights

The authors evaluated RaCT on three large-scale, standard benchmarks for collaborative filtering: MovieLens-20M (ML-20M), the Netflix Prize dataset, and the Million Song Dataset (MSD).

  • Consistent Boost: RaCT consistently improved the performance of VAEs across all three datasets. On ML-20M, it boosted NDCG@100 from 0.426 to 0.434.
  • Immediate Impact: Validation curves show that as soon as the RaCT "plug-in" is activated after MLE pre-training, there is a sharp, immediate jump in ranking performance.
  • Amortization is Key: Unlike traditional L2R methods that require expensive re-optimization for every new list, RaCT "amortizes" the cost. Once the Critic is trained, it provides an immediate gradient signal to the Actor.
  • Efficiency & Versatility: While more complex than a standard VAE, RaCT is significantly faster than traditional L2R methods like LambdaRank or WARP. Furthermore, RaCT isn't just for VAEs; it also improved Denoising Autoencoders (DAE) and Matrix Factorization (MF) models.

Limitations and Future Work

  • The "Power User" Ceiling: The authors noted that RaCT actually slightly hurt performance for users with extremely high activity levels (>750 interactions). They hypothesize this is due to a lack of training data for the Critic in that specific high-activity regime.
  • Better Features: The 3-dimensional feature vector for the Critic is simple. Future work could explore more complex "sufficient statistics" to give the Critic an even better understanding of the ranking landscape.
  • Beyond VAEs: While RaCT improves VAEs, some simple SVD-based models still perform very well. Integrating RaCT-style training into those linear models is a promising next step.

Summary updated

Abstract

We investigate new methods for training collaborative filtering models based on actor-critic reinforcement learning, to more directly maximize ranking-based objective functions. Specifically, we train a critic network to approximate ranking-based metrics, and then update the actor network to directly optimize against the learned metrics. In contrast to traditional learning-to-rank methods that require re-running the optimization procedure for new lists, our critic-based method amortizes the scoring process with a neural network, and can directly provide the (approximate) ranking scores for new lists. We demonstrate the actor-critic's ability to significantly improve the performance of a variety of prediction models, and achieve better or comparable performance to a variety of strong baselines on three large-scale datasets.
BibTeX
@inproceedings{lobel2020ract,
  author = {Sam Lobel and Chunyuan Li and Jianfeng Gao and Lawrence Carin},
  title = {{RaCT: Toward Amortized Ranking-Critical Training For Collaborative Filtering}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=1ZOVxbMwqcl},
}
Figure