Abstract
The ‘old world’ instrument, survey, remains a tool of choice for firms to obtain ratings of satisfaction and experience that customers realize while interacting online with firms. While avenues for survey have evolved from emails and links to pop-ups while browsing, the deficiencies persist. These include - reliance on ratings of very few respondents to infer about all customers’ online interactions; failing to capture a customer’s interactions over time since the rating is a one-time snapshot; and inability to tie back customers’ ratings to specific interactions because ratings provided relate to all interactions. To overcome these deficiencies we extract proxy ratings from clickstream data, typically collected for every customer’s online interactions, by developing an approach based on Reinforcement Learning (RL). We introduce a new way to interpret values generated by the value function of RL, as proxy ratings. Our approach does not need any survey data for training. Yet, on validation against actual survey data, proxy ratings yield reasonable performance results. Additionally, we offer a new way to draw insights from values of the value function, which allow associating specific interactions to their proxy ratings. We introduce two new metrics to represent ratings - one, customer-level and the other, aggregate-level for click actions across customers. Both are defined around proportion of all pairwise, successive actions that show increase in proxy ratings. This intuitive customer-level metric enables gauging the dynamics of ratings over time and is a better predictor of purchase than customer ratings from survey. The aggregate-level metric allows pinpointing actions that help or hurt experience. In sum, proxy ratings computed unobtrusively from clickstream, for every action, for each customer, and for every session can offer interpretable and more insightful alternative to surveys.
Introduction
Traditional customer satisfaction surveys are increasingly inadequate for the fast-paced digital economy. They suffer from extreme sampling bias, with response rates often falling below 1%, and provide only a static "snapshot" of user experience. This lack of granularity prevents firms from identifying specific pain points within a customer journey. This paper proposes a novel framework to extract "proxy ratings" directly from clickstream data using Reinforcement Learning (RL), enabling continuous, unobtrusive measurement for 100% of website visitors.
Methodology: RL-Based Proxy Ratings
The core innovation lies in modeling browsing behavior as a Reinforcement Learning problem. We utilize a Recurrent Neural Network (LSTM) to encode a user's action history into a state vector $S_t$. The model is trained using Temporal Difference (TD) learning, where the reward $r$ is binary: 1 for a purchase and 0 otherwise.
The value function $V(S_t)$, representing the expected future reward from a given state, is re-interpreted as a real-time proxy for customer satisfaction. To operationalize this, we introduce the Proportion of Good Ratings ($Z$), which measures how frequently successive actions lead to an increase in $V(S_t)$. This metric can be applied at the individual journey level ($Z^{(k)}$) or aggregated for specific action pairs ($Z(a_u, a_w)$).
Experimental Results
The model was tested on clickstream data from a major consumer electronics e-commerce site, specifically within the laptop category (~53,000 journeys).
Validation against Surveys
While the model requires no survey data for training, it was validated against a subset of ~8,500 journeys containing actual survey responses. The proxy ratings predicted human scores with an accuracy of 63-66% and an F1-score of 0.71-0.74.
Predictive Power and Metric Distribution
A key finding is that proxy ratings are significantly better predictors of purchase than actual surveys. Using the $Z^{(k)}$ metric to predict purchase yielded an AUC of 0.73, compared to just 0.51 for survey scores.
As illustrated in the histograms above, the distribution of $Z^{(k)}$ shifts as journey length $L$ increases, reflecting the evolving nature of customer engagement. This granularity allows for the identification of specific "pain points." For example, the transition from Home to ProductDetail (skipping the discovery funnel) often resulted in lower proxy ratings ($Z=0.22$) compared to the more natural Customize to ProductDetail transition ($Z=0.77$).
Conclusion
By leveraging the internal mechanics of Reinforcement Learning, we can transform raw clickstream data into a powerful, real-time instrument for measuring customer satisfaction. This "Survey without Questions" approach eliminates sampling bias and provides the granular insights necessary to optimize digital customer journeys. Future research will explore multi-goal identification to better capture the diverse intents of online visitors.
Summary updated