Abstract
The 'old world' instrument, survey, remains a tool of choice for firms to obtain ratings of satisfaction and experience that customers realize while interacting online with firms. While avenues for survey have evolved from emails and links to pop-ups while browsing, the deficiencies persist. These include - reliance on ratings of very few respondents to infer about all customers' online interactions; failing to capture a customer's interactions over time since the rating is a one-time snapshot; and inability to tie back customers' ratings to specific interactions because ratings provided relate to all interactions. To overcome these deficiencies we extract proxy ratings from clickstream data, typically collected for every customer's online interactions, by developing an approach based on Reinforcement Learning (RL). We introduce a new way to interpret values generated by the value function of RL, as proxy ratings. Our approach does not need any survey data for training. Yet, on validation against actual survey data, proxy ratings yield reasonable performance results. Additionally, we offer a new way to draw insights from values of the value function, which allow associating specific interactions to their proxy ratings. We introduce two new metrics to represent ratings - one, customer-level and the other, aggregate-level for click actions across customers. Both are defined around proportion of all pairwise, successive actions that show increase in proxy ratings. This intuitive customer-level metric enables gauging the dynamics of ratings over time and is a better predictor of purchase than customer ratings from survey. The aggregate-level metric allows pinpointing actions that help or hurt experience. In sum, proxy ratings computed unobtrusively from clickstream, for every action, for each customer, and for every session can offer interpretable and more insightful alternative to surveys.
Introduction
Traditional online surveys are increasingly inadequate for the modern digital landscape. They suffer from severe sampling bias, with response rates often below 1%, and provide only a blunt, session-level snapshot that fails to capture the dynamic nature of user experience. This paper addresses these challenges by proposing a method to measure user satisfaction unobtrusively for all users at the granularity of individual clicks.
Methodology: RL-Based Proxy Ratings
The core of the proposed approach is modeling user browsing as a decision-theoretic process using Reinforcement Learning (RL).
- State Representation: A Recurrent Neural Network (LSTM) encodes the sequence of past click actions into a continuous vector, capturing the session history.
- Reward Design: A sparse reward function is used, where a "Purchase" action yields a reward of 1, and all other actions yield 0.
- Value Iteration: Using model-free Temporal Difference (TD) learning, the algorithm estimates the value function $V(S_t)$—the expected future reward—for every state.
The primary innovation lies in interpreting the value function $V(S_t)$ as a "proxy rating" for user experience. To track satisfaction momentum, the authors introduce the Proportion of Good Ratings ($Z^{(k)}$), which measures the frequency of step-to-step increases in these proxy ratings.
Experimental Setup and Validation
The model was trained on clickstream data from a consumer electronics e-commerce site, comprising ~53,000 journeys. Critically, actual pop-up survey responses (collected from 0.7% of users) were strictly withheld during training and used only for validation. This allows for an objective assessment of whether RL-generated proxy ratings correlate with explicit human feedback.
Key Results
Survey Validation and Purchase Prediction
Without training on survey data, the RL proxy ratings predicted "poor" vs. "good" survey responses with 63-66% accuracy and an F1-score of up to 0.74. More importantly, the derived metric $Z^{(k)}$ proved to be a superior predictor of eventual purchases (AUC 0.73) compared to the users' own explicit survey scores (AUC 0.51).
Distribution of Satisfaction Momentum
The distribution of the $Z^{(k)}$ metric across different journey lengths ($L$) provides insights into how satisfaction momentum varies. As shown below, the metric remains a robust indicator across various session durations.
Action-Level Insights
By calculating proxy ratings at every click, the model can pinpoint UX bottlenecks. For instance, moving from a Customize page to a ProductDetail page showed a high proportion of good ratings (0.77), while regressive steps, such as moving back to a general ProductCategory page, showed significantly lower scores (0.44).
Conclusion
This research demonstrates that proxy ratings derived from clickstream data offer a more granular, dynamic, and predictive alternative to traditional surveys. While the current model assumes a single goal (purchase), future work could extend this to multi-goal environments, further refining the accuracy of satisfaction measurement for diverse user intents.
Summary updated