TL;DR
This paper introduces a novel four-phase model for understanding user intent on interactive systems by analyzing sequences of user actions. By leveraging Long Short-Term Memory (LSTM) networks, the model learns to predict user outcomes and identify specific actions that significantly influence these predictions, offering a path to optimize user experience.
What Problem Does This Paper Address?
In today's digital landscape, users interact with a vast array of web and mobile applications, performing actions that are often driven by an underlying intent. While user behavior can be complex and difficult to model, understanding user intent is crucial for designing more effective and user-friendly interactive systems. The challenge lies in translating the raw, sequential data of user actions (like clicks, scrolls, and taps) into meaningful insights about their goals. Existing methods often struggle to capture the dynamic nature of these interactions and their direct correlation with desired outcomes. This paper aims to bridge this gap by developing a system that can learn user intent from these action sequences, enabling objective maximization and system improvement.
Methods
The paper proposes a four-phase model:
- Data Generation: Raw user interaction data (clicks, scrolls, etc.) is collected and assembled into ordered sequences based on timestamps. These sequences are then processed to extract relevant features for each action event.
- Sequence Learning: Long Short-Term Memory (LSTM) neural networks are employed to learn representations of these action sequences. The LSTMs are trained to predict a specific outcome (e.g., purchase, conversion) using a binary cross-entropy loss function. This phase results in a pre-trained LSTM model (LSTMT).
- Objective Maximization: The LSTMT model is used to evaluate new action sequences. By comparing predicted outcomes with actual outcomes, a confusion matrix is built. Sequences where predictions differ from reality are clustered to identify groups of actions associated with similar intents.
- Sequence Analysis & Semi-Automated Contrasting: This phase analyzes the clustered intent-specific sessions. It evaluates each time step in an action sequence to observe how predictions change. By measuring the distance between predictions at consecutive timesteps, the model identifies actions that cause significant shifts in predicted intent. These "impactful" sequences are then analyzed by a human expert to identify system features that might be causing these changes, leading to actionable insights for system improvement.
Datasets
The paper describes evaluation on data collected from a ticketing website where users can sell or purchase tickets. The data was anonymized, and each user session was treated as an independent entity. Sessions were sampled from three months in late 2016 for training and from months in early 2016 and early 2017 for testing.
Key Insights and Ideas
- Intent is Learnable from Action Sequences: The core idea is that user intent, while not directly observable, can be inferred by analyzing the temporal patterns of their actions within an interactive system.
- LSTMs are Powerful for Sequential Intent: LSTMs are well-suited for capturing the dependencies and nuances within user action sequences, enabling them to learn complex relationships between actions and outcomes.
- Focus on Outcome Prediction for Intent Inference: Instead of directly predicting intent, the model uses observable outcomes (like purchases) as labels to train sequence learners, which then implicitly reveal intent patterns.
- Identifying "Impactful" Actions: The sequence analysis phase is crucial for pinpointing specific actions that cause significant deviations between predicted and actual outcomes, highlighting areas for system optimization.
- Hybrid Human-AI Approach: The semi-automated contrasting phase emphasizes the value of combining machine learning's pattern recognition with human expertise for effective system improvement.
Key Results
The neural model, when evaluated on predicting conversion probability within a session, achieved:
- An average accuracy of 0.89 and an average recall of 0.85 when evaluated at the penultimate step (k=1) before the final outcome.
- A significant monotonic improvement in prediction accuracy as the model got closer to the end of the session. Recall at k=4 was 0.70, rising to 0.81 at k=2, and 0.85 at k=1.
- This improvement demonstrates the model's ability to leverage local action context for better intent prediction.
Limitations and Future Work
- Semi-Automated Contrasting: The final phase of contrasting is currently performed semi-automatically by a human expert. The authors suggest that further automation of this phase could enhance efficiency.
- Behavior vs. Intent: While the model focuses on intent, the authors acknowledge that understanding user behavior is also important. They suggest that their models could benefit future studies in behavior modeling by identifying more specific attributes of users and sessions.
- Active vs. Passive Systems: The paper discusses both active (real-time prediction) and passive (dashboard analysis) deployment scenarios, indicating flexibility but also potential for further exploration in how these modes can be integrated.
- Generalizability: While the model is presented as system-independent, its effectiveness across a wider range of diverse interactive systems and objectives would be a valuable area for future research.
Summary updated