State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study

Jin Huang, Harrie Oosterhuis, Bunyamin Cetinkaya, Thijs Rood, Maarten de Rijke

TL;DR

This reproducibility study re-examines the claim that attention-based state encoders are the "gold standard" for Reinforcement Learning-based Recommender Systems (RL4Rec). While the authors successfully replicate the original findings on the Yahoo! R3 dataset, they discover that these results do not generalize to other datasets (like the Coat shopping dataset), where simpler architectures like Multi-Layer Perceptrons (MLPs) can actually outperform complex attention mechanisms.

What Problem Does This Paper Address?

In RL4Rec, a "state encoder" is responsible for turning a user’s messy history of interactions into a mathematical representation (a dense vector) that the RL agent can use to make the next recommendation. A previous influential study by Liu et al. (2020) compared several encoders and concluded that an attention-based approach was superior.

However, the authors of this paper identified several gaps in that original research:

  1. Bias: The original evaluation used simulators based on logged user data, which is often riddled with "popularity bias" (users seeing what’s popular, not necessarily what they like).
  2. Limited Scope: It only tested one RL method (actor-critic) and a small handful of encoders.
  3. Generalization: It wasn't clear if the "attention is best" rule held true across different types of datasets.

Methods

The researchers used the SOFA (Simulator for OFfline leArning and evaluation) framework. Unlike standard simulators, SOFA uses Inverse Propensity Scoring (IPS) to debias the data, creating a more "honest" environment for the RL agent to learn user preferences.

They compared seven different state encoders:

  • The Original Four: BOI (Bag of Items), PLD (Pairwise Local Dependency), Avg (Simple Average), and Attention.
  • Three New Additions: MLP (Multi-Layer Perceptron), GRU (Gated Recurrent Unit), and CNN (Convolutional Neural Network).

For the RL "brain," they swapped the original actor-critic method for a Deep Q-Network (DQN), which is the most popular RL method in recommendation research due to its relative simplicity and efficiency.

Datasets

The study utilized two key datasets that are unique because they contain both biased "natural" user logs and unbiased "randomly sampled" test sets:

  • Yahoo! R3: A large music rating dataset (15,400 users, 1,000 items).
  • Coat: A smaller shopping dataset (290 users, 300 items).

Key Insights and Ideas

  • Attention isn't a Silver Bullet: On the Yahoo! R3 dataset, the attention-based encoder did indeed perform best. However, on the Coat dataset, it performed worse than a simple average or a standard MLP.
  • Dataset Size Matters: The authors speculate that complex encoders like Attention and GRU require more data to shine. On smaller datasets like Coat, these complex models may overfit or fail to converge to an optimal policy, whereas simpler models generalize better.
  • The "Aha" Moment for MLPs: The study found that simply adding a non-linear activation function (like ReLU or Tanh) to a basic "Average" encoder—effectively turning it into an MLP—can lead to massive performance gains, sometimes even beating the most complex architectures.
  • Computational Trade-offs: Attention encoders are significantly more "expensive" to train. On Yahoo! R3, the Attention encoder took ~30 seconds per 1,000 steps, compared to just ~8 seconds for the Average encoder. If a simpler MLP performs similarly or better, the extra compute for attention isn't justified.

Key Results

  • Yahoo! R3 Results: Replicated the original study. Attention and GRU were the top performers, significantly beating BOI and PLD.
  • Coat Results: The MLP encoder was the clear winner, significantly outperforming the Attention encoder.
  • Convergence: Simpler encoders (BOI, PLD) converge very quickly but to a lower "ceiling" of performance. MLPs converge faster than simple Averages, suggesting that non-linearity helps the agent learn the value of states more efficiently.
  • Activation Sensitivity: The choice of activation function for MLPs is critical. Tanh worked best for the Coat dataset, while ReLU was superior for Yahoo! R3.

Limitations and Future Work

The authors acknowledge that their study is limited to the DQN algorithm. While DQN is a standard benchmark, RL4Rec is moving toward more complex policy gradient methods like REINFORCE or newer actor-critic variants. Future work should investigate if the choice of the RL algorithm changes which state encoder is optimal. Additionally, the study highlights a desperate need for more public datasets that include the "randomly sampled" data required to build debiased simulators like SOFA.

Summary updated

Abstract

Methods for reinforcement learning for recommendation (RL4Rec) are increasingly receiving attention as they can quickly adapt to user feedback. A typical RL4Rec framework consists of (1) a state encoder to encode the state that stores the users' historical interactions, and (2) an RL method to take actions and observe rewards. Prior work compared four state encoders in an environment where user feedback is simulated based on real-world logged user data. An attention-based state encoder was found to be the optimal choice as it reached the highest performance. However, this finding is limited to the actor-critic method, four state encoders, and evaluation-simulators that do not debias logged user data. In response to these shortcomings, we reproduce and expand on the existing comparison of attention-based state encoders (1) in the publicly available debiased RL4Rec SOFA simulator with (2) a different RL method, (3) more state encoders, and (4) a different dataset. Importantly, our experimental results indicate that existing findings do not generalize to the debiased SOFA simulator generated from a different dataset and a Deep Q-Network (DQN)-based method when compared with more state encoders.
BibTeX
@inproceedings{huang2022state,
  author = {Jin Huang and Harrie Oosterhuis and Bunyamin Cetinkaya and Thijs Rood and Maarten de Rijke},
  title = {{State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2022},
  doi = {10.1145/3477495.3531716},
}

Loading related content...

Figure