Traditional recommendation systems are often "myopic," focusing on predicting the immediate probability of a click or purchase. However, a user’s interaction with a platform is not a series of independent events, but a continuous journey where each recommendation influences the user's future state. Recent research has shifted toward modeling these interactions as sequential decision-making processes, using Reinforcement Learning (RL) to maximize long-term rewards such as cumulative engagement, lifetime value, and even societal welfare.
The Markov Decision Process for Recommendations
The core shift in this field is reframing recommendation as a Markov Decision Process (MDP). In this framework, the system acts as an agent, the user represents the environment, and the "state" captures the user’s evolving preferences and interaction history.
By treating the interaction as a sequence, models can account for temporal dependencies that greedy models ignore. For instance, Shao et al. propose the SPLIT model, which uses RL to adaptively decompose a single interaction history into multiple "preference threads" (e.g., separating electronics shopping from baby gear). Similarly, Liu argues that session-based MDPs capture evolving intent more accurately than item-wise models, allowing for dynamic adjustments in multi-task learning.
From Learning to Planning
When interactions are sequential, the task moves from simple classification to planning. Instead of asking "What song does the user want now?", the system asks "What sequence of songs will maximize their satisfaction over the next hour?"
Liebman et al. demonstrate this by applying Monte Carlo Tree Search (MCTS) to playlist generation, optimizing for sequence-level objectives rather than individual item preferences. In the context of "infinite scroll" interfaces, Zhao et al. frame recommendation as a Stochastic Shortest Path (SSP) problem. Their approach explicitly balances the reward of an immediate click against the risk of the user reaching an "absorbing state"—terminating the session.
Optimizing for Life-Time Value
A major motivation for sequential modeling is the optimization of Life-Time Value (LTV). Myopic models may prioritize "clickbait" that provides immediate rewards but leads to long-term churn.
Theocharous et al. show that RL can optimize for cumulative rewards across multiple sessions, using High-Confidence Off-Policy Evaluation to ensure these policies are safe to deploy. This long-term perspective is also critical in advertising; Hao et al. treat multi-channel bidding as a Dynamic Knapsack Problem, maximizing revenue across a user's entire conversion journey rather than treating each ad auction in isolation.
Beyond Clicks: Welfare and Manipulation
Sequential modeling also allows researchers to address the broader consequences of recommendation. Tabibian et al. introduce consequential ranking algorithms that use MDPs to balance immediate utility with long-term societal welfare, such as reducing the spread of toxic content.
However, the power to influence future states also carries risks. Acemoğlu et al. provide an economic model of behavioral manipulation, showing how AI can exploit "glossy" features to lead users toward low-quality products over time. This highlights the importance of aligning sequential agents with true user utility rather than just engagement metrics.
Modern Frontiers: Continuous Control and Dialogue
As the field matures, the mechanisms for sequential optimization are becoming more sophisticated:
- Continuous Control: Rather than picking discrete items, Wang et al. treat recommendation as a continuous control problem, navigating a "user preference space" to identify the optimal direction for interest shifts.
- Deep Exploration: Zhu et al. argue for "deep exploration," where agents strategically probe users over multiple interactions to uncover latent preferences that myopic strategies would never find.
- Personalized Alignment: In the era of Large Language Models, Zhao et al. use RL to help models dynamically infer user profiles through multi-turn dialogue, treating conversation as a sequential decision task.
By moving from static ranking to autonomous, decision-making agents—as discussed by Cai et al.—the research direction aims to create systems that don't just react to users, but actively navigate the long-term dynamics of human-AI interaction.
Key Foundational Work
An MDP-based Recommender System (2002) — contributed 2 concept elements to this direction.
Interactive exploratory search for multi page search results (2013) — contributed 1 concept elements to this direction.