The anatomy of a click: modeling user behavior on web information systems

Kunal Punera, Srujana MeruguCIKM 2010 · DOI

TL;DR

This paper introduces a dynamic Bayesian network (DBN) to model user behavior on web information systems, going beyond observed actions to infer hidden user intentions. By incorporating both observed and unobserved actions, and using a novel bi-clustering approach to account for user and task heterogeneity, the model achieves superior performance in predicting user session completions and post-session actions.

What Problem Does This Paper Address?

Understanding user intent is crucial for optimizing web information systems, from search engines to content portals. While observed user actions like clicks provide some insight, they are often insufficient to fully grasp a user's underlying goal. This is due to several factors:

  • Limited Observability: Web interfaces often simplify interactions, capturing only a subset of user actions (e.g., clicks, queries) while missing others like eye movements, scrolling, or result examination.
  • Ambiguity: Queries can be ambiguous (e.g., "greyhound" could refer to a dog or a bus), and even with clear queries, users might have diverse intents (e.g., finding information vs. making a purchase).
  • Heterogeneity: Users exhibit diverse behaviors based on their individual characteristics (e.g., age, tech-savviness) and the specific task they are trying to accomplish.

Existing models often struggle to capture this complexity, relying on simplistic assumptions like top-down result examination or treating all users and tasks uniformly. This paper aims to bridge this gap by developing a more comprehensive model that accounts for both observable and unobservable user actions, and addresses the inherent heterogeneity in user behavior.

Methods

The core of the paper is a dynamic Bayesian network (DBN) designed to model user sessions. Key aspects of their approach include:

  • Hidden Actions: The model explicitly incorporates "hidden" user actions (e.g., examining a result, skipping results) that are not directly observed by the system but are crucial for inferring intent.
  • State Representation: User states are represented by a vector of attributes, including both observed (e.g., number of clicks) and hidden (e.g., number of results examined) components.
  • Sequential Modeling: The model captures the temporal dependencies between user states and actions using Markovian assumptions, allowing for the prediction of future behavior based on past interactions.
  • Bi-Clustering: To address user and task heterogeneity, the authors propose a novel bi-clustering algorithm. This algorithm simultaneously partitions users and tasks into groups, learning separate DBN models for each user-task bicluster. This allows for more personalized and task-specific behavior modeling.
  • Learning and Inference: The model parameters are learned by maximizing the likelihood of observed data, while user intent is inferred by finding the most probable sequence of hidden actions given a partial observation of the session.

Datasets

The authors use a real-world web search log dataset for their experiments. This dataset comprises approximately 210,000 sessions from around 3,500 users over a month in the fall of 2008. The data includes queries, viewed/clicked results, and timestamps. For evaluation, they also created a subset of non-navigational queries by removing navigational ones from the original dataset to better isolate the model's performance on more complex search tasks.

Key Insights and Ideas

  • Hidden Actions Matter: The most significant insight is that modeling unobserved user actions (like result examination and skipping) is critical for accurately inferring user intent, especially in the early stages of a search session.
  • Heterogeneity is Key: Treating all users and tasks the same leads to suboptimal performance. The bi-clustering approach, which learns separate models for different user-task groups, significantly improves prediction accuracy.
  • Early Session Prediction is Hardest (and Most Valuable): The model's advantage over baselines is most pronounced when very little information (few clicks) about the session is available. This suggests the model is particularly effective at making predictions when they are most needed.
  • User Behavior is Richer Than Simple Click Streams: The paper challenges the assumption of simple top-down result examination, highlighting the importance of modeling more complex interaction patterns observed in eye-tracking studies.

Key Results

The proposed model, ALLACTSCLUSTS, consistently outperforms baselines in predicting session completions and post-session actions.

  • Session Completion:
    • NUMCLICKS: Lower values indicate better performance. ALLACTSCLUSTS achieved significantly better NUMCLICKS scores than the baseline ONLYNEXTACT (which only models "examine next" action and no clustering). For example, with 0 observed clicks, ALLACTSCLUSTS had a NUMCLICKS of 0.81 compared to 0.87 for ONLYNEXTACT.
  • EDITDISTANCE: Lower values indicate better performance. ALLACTSCLUSTS also outperformed ONLYNEXTACT on EDITDISTANCE. For 0 observed clicks, ALLACTSCLUSTS had an EDITDISTANCE of 4.05 compared to 4.33 for ONLYNEXTACT.
  • Post-Session Actions:
    • Classification Accuracy: ALLACTSCLUSTS achieved a classification accuracy of > 65% for predicting post-session actions (e.g., leaving the session, reformulating query) even when zero clicks had been observed. This significantly outperforms a baseline classifier (BASECLASS) that only uses observed features, which achieved around 33% accuracy in this scenario.
  • Clustering Benefits: The paper also shows that the clustering component is crucial. A version of their model without clustering (ONECLUST) performed similarly to ALLACTSCLUSTS when many clicks were observed, but significantly worse when zero clicks were observed, highlighting the value of clustering for initial predictions.

Limitations and Future Work

  • Computational Feasibility: While the model is powerful, the paper acknowledges that estimating the posterior distribution of hidden variables can be computationally intensive, requiring careful optimization strategies.
  • Ground Truth: The evaluation relies on observed click data, and the authors note that ideal evaluation would involve ground truth from eye-tracking studies.
  • Generalizability: While the model is presented as general, the primary instantiation and evaluation are on web search. Extending it to other complex interactive systems (e.g., social media, e-commerce) would be valuable.
  • Future Work: The authors suggest exploring more sophisticated clustering techniques, incorporating richer domain knowledge, and investigating different optimization criteria for system response. They also mention the potential for applying the abstract model to multi-frame/multi-tabbed views and long-term offline interactions.

Summary updated

BibTeX
@inproceedings{punera2010anatomy,
  author = {Kunal Punera and Srujana Merugu},
  title = {{The anatomy of a click: modeling user behavior on web information systems}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2010},
  doi = {10.1145/1871437.1871563},
}
Figure