Deep Reinforcement Learning for Dynamic Recommendation with Model-agnostic Counterfactual Policy Synthesis

Siyu Wang, Xiaocong Chen, Lina Yao, Julian J. McAuley

Abstract

Recent advances in recommender systems have proved the potential of Reinforcement Learning (RL) to handle the dynamic evolution processes between users and recommender systems. However, learning to train an optimal RL agent is generally impractical with commonly sparse user feedback data in the context of recommender systems. To circumvent the lack of interaction of current RL-based recommender systems, we propose to learn a general Model-Agnostic Counterfactual Synthesis (MACS) Policy for counterfactual user interaction data augmentation. The counterfactual synthesis policy aims to synthesise counterfactual states while preserving significant information in the original state relevant to the user's interests, building upon two different training approaches we designed: learning with expert demonstrations and joint training. As a result, the synthesis of each counterfactual data is based on the current recommendation agent's interaction with the environment to adapt to users' dynamic interests. We integrate the proposed policy with Deep Deterministic Policy Gradient (DDPG), Soft Actor Critic (SAC), and Twin Delayed DDPG in an adaptive pipeline. The empirical results on both online simulation and offline datasets demonstrate the effectiveness and generalisation of our counterfactual synthesis policy and verify that it improves the performance of RL recommendation agents.

Introduction

Reinforcement Learning (RL) is a powerful tool for recommendation because it can model the dynamic, evolving nature of user interests. However, RL agents are notoriously data-hungry. In the real world, user feedback is incredibly sparse—users only interact with a tiny fraction of available items. When an RL agent has limited interaction data, it struggles to "grasp" a user's true preferences. If a specific situation hasn't been recorded, the system might mistakenly assume a lack of interest, hampering its ability to provide personalized suggestions. Traditional data augmentation (like adding random noise) often fails here because it doesn't respect the underlying causal relationships that drive user behavior.

Methodology: The MACS Framework

The proposed Model-Agnostic Counterfactual Synthesis (MACS) framework introduces a "plug-and-play" approach to solve the data sparsity problem by generating synthetic but causally valid "counterfactual" user interaction data.

State Decomposition and Causal Intervention

The core idea is that a user's "state" (history, demographics, etc.) consists of essential components (which directly reflect their core interests) and trivial components (less relevant details). MACS acts as a secondary policy that learns to intervene on the "trivial" parts of a state to generate a new, "counterfactual" state ($S_c$) that is slightly different from the original ($S_t$).

Causal diagrams illustrating (a) a standard MDP, (b) state decomposition into essential and trivial components, and (c) the counterfactual intervention process.

To ensure the synthesis is valid, the authors use do-calculus and KL-Divergence. If the predicted reward distribution for the synthetic state matches the original state's reward distribution, it implies that only "trivial" components were changed, and the "essential" user interest was preserved.

Training and Augmentation Pipeline

The framework supports two primary training strategies for the synthesis policy:

  1. Learning with Expert Demonstrations: A pre-trained "expert" policy acts as a teacher to guide the synthesis.
  2. Joint Training: The recommendation policy and the counterfactual synthesis policy are trained together in an adaptive pipeline.
Overview of the MACS framework, illustrating the training procedure for the counterfactual synthesis policy and the subsequent data augmentation process.

Because the reward function is the primary signal, MACS is model-agnostic—it can be integrated into various RL architectures like DDPG, SAC, or TD3.

Experimental Evaluation

The researchers evaluated their approach using both online simulations and offline benchmarks:

  • Online Simulation: VirtualTaobao, a simulator trained on hundreds of millions of real-world data points to mimic live retail environments.
  • Offline Datasets: MovieLens (100k and 1M), Douban-Book, and BookCrossing. These datasets vary in sparsity, allowing for a robust test of the model's ability to handle limited data.

Key Results

Empirical results show that MACS consistently improves the performance of underlying RL models:

  • Click-Through Rate (CTR): In VirtualTaobao, DDPG equipped with MACS showed a dramatic and steady increase in CTR compared to vanilla DDPG, which plateaued much earlier.
  • Accuracy and Recall: On MovieLens-1M, MACS-enhanced models achieved significantly higher metrics. For instance, DDPG-E reached an accuracy of 0.8035 compared to the baseline DDPG's 0.5820.
  • Sparsity Handling: Improvements were particularly pronounced on the sparser BookCrossing and Douban-Book datasets, proving that counterfactual augmentation is an effective antidote to data scarcity.

Discussion and Conclusion

MACS demonstrates that causality-aware synthesis is superior to random data augmentation. By generating "what if" scenarios, the system can effectively explore the state space without needing real user interactions for every possible sequence of actions.

However, limitations remain. Training a secondary synthesis policy increases computational overhead. Furthermore, the current framework identifies essential components implicitly through the reward function; future work could involve more explicit disentanglement of these features. Success also depends on hyperparameter sensitivity, specifically the threshold for when the recommendation policy is mature enough to begin training the synthesizer.

Summary updated

Abstract

Recent advances in recommender systems have proved the potential of Reinforcement Learning (RL) to handle the dynamic evolution processes between users and recommender systems. However, learning to train an optimal RL agent is generally impractical with commonly sparse user feedback data in the context of recommender systems. To circumvent the lack of interaction of current RL-based recommender systems, we propose to learn a general Model-Agnostic Counterfactual Synthesis (MACS) Policy for counterfactual user interaction data augmentation. The counterfactual synthesis policy aims to synthesise counterfactual states while preserving significant information in the original state relevant to the user's interests, building upon two different training approaches we designed: learning with expert demonstrations and joint training. As a result, the synthesis of each counterfactual data is based on the current recommendation agent's interaction with the environment to adapt to users' dynamic interests. We integrate the proposed policy Deep Deterministic Policy Gradient (DDPG), Soft Actor Critic (SAC) and Twin Delayed DDPG in an adaptive pipeline with a recommendation agent that can generate counterfactual data to improve the performance of recommendation. The empirical results on both online simulation and offline datasets demonstrate the effectiveness and generalisation of our counterfactual synthesis policy and verify that it improves the performance of RL recommendation agents.
BibTeX
@article{wang2025deep,
  author = {Siyu Wang and Xiaocong Chen and Lina Yao and Julian J. McAuley},
  title = {{Deep Reinforcement Learning for Dynamic Recommendation with Model-agnostic Counterfactual Policy Synthesis}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2025},
  doi = {10.1109/tnnls.2023.3329808},
}
Figure