Introduction: The Myopia of Modern Recommenders
In the era of "infinite scroll" feeds—seen on platforms like TikTok, Amazon, and Taobao—the primary goal of a recommendation system is to maximize total user engagement over an entire session. However, most industrial recommenders remain "myopic," optimizing for immediate metrics like Click-Through Rate (CTR) for a single set of items. This approach often ignores the long-term consequences of a recommendation: a "clickbaity" item might garner an immediate hit but frustrate the user enough to quit the session entirely.
To address this, researchers from Alibaba and Microsoft have reframed sequential recommendation as a Stochastic Shortest Path (SSP) problem. By explicitly modeling the trade-off between immediate rewards and the probability of a user leaving the platform, they aim to maximize the cumulative volume of engagement across the full browsing sequence.
The MDP-SSP Framework: Modeling the Long Game
The core of the proposed approach is the MDP-SSP framework, which treats the recommendation process as a personalized Markov Decision Process (MDP). In this model:
- States ($s_t$): Represent the current step in the user's browsing sequence.
- Actions ($a_t$): The specific items recommended to the user.
- Rewards ($r_t$): The immediate engagement generated (e.g., a click or page view).
- Transitions: The probability that a user either continues to the next state ($s_{t+1}$) or enters an "absorbing state" ($s_A$), representing the moment they quit the application.
By defining the user's departure as the end of the path, the optimization goal becomes finding a sequence of actions that maximizes the expected total reward before reaching the absorbing state. Because the browsing sequence is strictly forward-moving in time, the transition matrix possesses a special structure that allows for highly efficient solving via Dynamic Programming (backwards induction).
Predicting the "Quit" Signal
A significant challenge in this framework is predicting the "quit probability" for individual items. Since we only observe when a user leaves after a sequence of items, it is difficult to pinpoint which specific item triggered the exit. The authors utilized Multi-Instance Learning (MIL), specifically MI-SVM, to estimate these probabilities from "bags" of items viewed in historical sessions.
Furthermore, the researchers emphasized the necessity of calibration. Raw model outputs are often biased and do not represent true probabilities. By applying Platt scaling, they ensured that predicted quit scores accurately reflected real-world departure rates, a step critical for the mathematical integrity of the MDP cumulative calculations.
Experimental Results and Real-World Impact
The framework was rigorously tested using both offline simulations and a massive live A/B test involving approximately 10 million users daily on a major e-commerce platform.
In the live production environment, the MDP-SSP framework achieved a 7% increase in cumulative clicks (Item Page Views) compared to the existing production baseline. This result confirms that "planning" for the session length yields significantly better outcomes than a greedy, CTR-only approach.
Offline evaluations further highlighted the robustness of the SSP method. As shown in the simulation results, the SSP approach consistently outperformed both "Greedy" ranking and "Beam Search" heuristics. While the SSP method is more sensitive to prediction noise than simpler greedy strategies, it maintains a substantial performance lead even under moderate levels of model uncertainty.
Conclusion and Future Directions
The MDP-SSP framework represents a shift from myopic ranking to strategic session planning in recommendation systems. By mathematically balancing immediate engagement with session longevity, it provides a principled way to increase total user value.
One remaining challenge is the "curse of dimensionality" associated with item deduplication. Currently, tracking which items have already been shown to prevent repeats can cause the state space to explode. While the authors implemented a compromise strategy for this study, integrating deduplication constraints directly into the optimization framework remains a primary focus for future research.
Summary updated