State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study

Jin Huang, Harrie Oosterhuis, Bunyamin Cetinkaya, Thijs Rood, Maarten de Rijke

Abstract

Methods for reinforcement learning for recommendation (RL4Rec) are increasingly receiving attention as they can quickly adapt to user feedback. A typical RL4Rec framework consists of (1) a state encoder to encode the state that stores the users' historical interactions, and (2) an RL method to take actions and observe rewards. Prior work compared four state encoders in an environment where user feedback is simulated based on real-world logged user data. An attention-based state encoder was found to be the optimal choice as it reached the highest performance. However, this finding is limited to the actor-critic method, four state encoders, and evaluation-simulators that do not debias logged user data. In response to these shortcomings, we reproduce and expand on the existing comparison of attention-based state encoders (1) in the publicly available debiased RL4Rec SOFA simulator with (2) a different RL method, (3) more state encoders, and (4) a different dataset. Importantly, our experimental results indicate that existing findings do not generalize to the debiased SOFA simulator generated from a different dataset and a Deep Q-Network (DQN)-based method when compared with more state encoders.

Introduction

In Reinforcement Learning-based Recommender Systems (RL4Rec), the "state encoder" plays a critical role. It is responsible for transforming a user’s complex history of interactions into a dense mathematical representation—a state vector—that the RL agent uses to determine the next best recommendation.

A landmark study by Liu et al. (2020) previously established attention-based encoders as the superior choice for this task. However, that research left several questions unanswered regarding the impact of data bias, the choice of RL algorithm, and the generalizability of the results across different datasets. This study reproduces the original findings while expanding the scope to include debiased simulators, additional encoder architectures, and diverse datasets to determine if attention truly remains the "gold standard."

Methodology and Experimental Design

To address the limitations of previous work, this study utilized the SOFA (Simulator for OFfline leArning and evaluation) framework. SOFA is distinct because it employs Inverse Propensity Scoring (IPS) to debias logged user data. This creates a more accurate environment for the RL agent by accounting for "popularity bias," where users are often shown popular items rather than those that align strictly with their personal preferences.

State Encoders

The researchers compared seven distinct state encoder architectures:

  • Original Encoders: Bag of Items (BOI), Pairwise Local Dependency (PLD), Simple Average (Avg), and Attention.
  • New Additions: Multi-Layer Perceptron (MLP), Gated Recurrent Unit (GRU), and Convolutional Neural Network (CNN).

RL Algorithm and Data

While the original study focused on actor-critic methods, this reproduction uses a Deep Q-Network (DQN), currently the most prevalent RL method in recommendation research due to its efficiency. The experiments were conducted on two datasets that provide both biased natural logs and unbiased random test sets:

  1. Yahoo! R3: A large-scale music rating dataset (15,400 users, 1,000 items).
  2. Coat: A smaller shopping dataset (290 users, 300 items).

Key Findings

1. The Limits of Attention

The study successfully replicated the original findings on the Yahoo! R3 dataset, where the attention-based encoder and the GRU were the top performers. However, these results failed to generalize to the Coat dataset. In the smaller Coat environment, the attention mechanism performed worse than a simple average or a standard MLP.

2. Dataset Size and Model Complexity

The results suggest that complex encoders like Attention and GRU require significant amounts of data to reach their potential. On smaller datasets like Coat, these high-capacity models may overfit or fail to converge to an optimal policy. Conversely, simpler models demonstrate better generalization in data-constrained environments.

3. The Efficiency of MLPs

A significant insight from the study is the effectiveness of the MLP encoder. By simply adding a non-linear activation function (such as ReLU or Tanh) to a basic "Average" encoder, the researchers observed massive performance gains. On the Coat dataset, the MLP was the clear winner, outperforming more complex architectures while requiring significantly less computational overhead. For instance, on Yahoo! R3, the Attention encoder took approximately 30 seconds per 1,000 steps, whereas the Average-based encoders took only 8 seconds.

4. Convergence and Sensitivity

Simpler encoders like BOI and PLD converge rapidly but reach a lower performance ceiling. MLPs converge faster than simple Averages, indicating that non-linearity helps the agent learn state values more efficiently. Furthermore, the choice of activation function is critical: Tanh proved most effective for the Coat dataset, while ReLU was superior for Yahoo! R3.

Conclusion and Future Work

This reproducibility study demonstrates that there is no single "best" state encoder for RL4Rec. While attention-based mechanisms excel in large-scale environments, simpler MLP-based architectures are often more effective and efficient for smaller datasets.

Future research should explore whether these findings hold across different RL algorithms beyond DQN, such as REINFORCE or modern actor-critic variants. Additionally, the study underscores the necessity for more public datasets containing randomly sampled data to facilitate the development of debiased simulators and more robust RL4Rec benchmarks.

Summary updated

Abstract

Methods for reinforcement learning for recommendation (RL4Rec) are increasingly receiving attention as they can quickly adapt to user feedback. A typical RL4Rec framework consists of (1) a state encoder to encode the state that stores the users' historical interactions, and (2) an RL method to take actions and observe rewards. Prior work compared four state encoders in an environment where user feedback is simulated based on real-world logged user data. An attention-based state encoder was found to be the optimal choice as it reached the highest performance. However, this finding is limited to the actor-critic method, four state encoders, and evaluation-simulators that do not debias logged user data. In response to these shortcomings, we reproduce and expand on the existing comparison of attention-based state encoders (1) in the publicly available debiased RL4Rec SOFA simulator with (2) a different RL method, (3) more state encoders, and (4) a different dataset. Importantly, our experimental results indicate that existing findings do not generalize to the debiased SOFA simulator generated from a different dataset and a Deep Q-Network (DQN)-based method when compared with more state encoders.
BibTeX
@inproceedings{huang2022state,
  author = {Jin Huang and Harrie Oosterhuis and Bunyamin Cetinkaya and Thijs Rood and Maarten de Rijke},
  title = {{State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2022},
  doi = {10.1145/3477495.3531716},
}
Figure