Learning Efficient Representations of Mouse Movements to Predict User Attention

Ioannis Arapakis, Luis A. Leiva

TL;DR

This paper explores using neural networks to predict user attention on search engine results pages (SERPs) by analyzing raw mouse cursor movements, moving beyond traditional handcrafted features. They demonstrate that both recurrent (RNNs) and convolutional neural networks (CNNs) can effectively learn representations from time-series and image-based encodings of mouse data, with CNNs generally outperforming RNNs.

What Problem Does This Paper Address?

Modern search engine results pages (SERPs) are increasingly complex, featuring not just traditional text links but also various "direct displays" like image carousels, videos, and advertisements. Understanding which of these elements capture user attention is crucial for search engines to optimize ranking, improve UI design, and enhance advertising effectiveness. While mouse cursor tracking offers a scalable and non-intrusive way to infer user attention compared to eye-tracking, previous research has relied heavily on manual feature engineering. This process is time-consuming, requires domain expertise, and may not capture the full complexity of user interaction. This paper aims to automate this process by leveraging deep learning to learn efficient representations of mouse movements for attention prediction.

Methods

The authors investigate two main types of neural network architectures: Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs).

  • RNNs: They explore standard RNNs, Long Short-Term Memory (LSTM) networks, Gated Recurrent Units (GRUs), and Bidirectional LSTMs (BLSTMs). These models are trained on raw mouse cursor data represented as multivariate time series of 2D coordinates. To handle variable-length sequences, they are padded or truncated to a fixed length of 50 timesteps.
  • CNNs: For CNNs, they employ transfer learning, using pre-trained models (AlexNet, SqueezeNet, ResNet50, VGG19) trained on ImageNet. The mouse cursor data is transformed into various visual representations:
  • Heatmaps: Representing cursor position density using 2D Gaussian kernels.
  • Trajectories: Lines connecting consecutive cursor points, with variations in color and line thickness to encode temporal information. These visual representations are rendered as images and fed into the CNNs.

The core idea is to compare the performance of these different network architectures and data representations in predicting whether a user noticed an ad.

Datasets

The study utilized a custom dataset collected through an online user study conducted via the Figure Eight crowdsourcing platform.

  • Data Collection: Participants performed transactional search tasks using Google Search. Mouse cursor movements and associated metadata (position, timestamp, event name, DOM element) were captured using custom JavaScript code (EVTRACK).
  • Ground Truth: After completing tasks, participants provided self-reported labels on a 5-point Likert scale indicating their attention to ads. These were converted into binary labels (noticed/not noticed).
  • Dataset Size: The final dataset comprised 2,289 search sessions with 45,082 mouse cursor positions. The data was split into training, validation, and test sets using a 60-10-30 ratio, with stratified sampling to maintain class distribution.

Key Insights and Ideas

  • Raw Data is Powerful: Neural networks can learn effective representations directly from raw mouse cursor data, eliminating the need for manual feature engineering and domain expertise. This significantly simplifies the process of building attention prediction models.
  • CNNs Excel at Visual Representations: CNNs, particularly when applied to image-based representations of mouse trajectories (like colored trajectories with variable line thickness), generally outperform RNNs. This suggests that visual encodings can capture complex spatio-temporal patterns effectively.
  • Temporal Encoding Matters: Representations that encode temporal information (e.g., trajectory color, line thickness) tend to perform better, highlighting the importance of capturing the sequence and duration of mouse movements.
  • Ad Placeholder Impact Varies: The presence of an ad placeholder in visual representations had a mixed effect. It significantly improved performance for left-aligned direct display ads but showed no significant change for organic or right-aligned ads. This suggests that the visual context of the ad's placement influences how effectively its presence can be encoded.
  • Shallow CNNs Can Be Competitive: Simpler CNN architectures like AlexNet and SqueezeNet achieved performance comparable to deeper models like ResNet50 and VGG19, indicating that efficient and less computationally intensive models can be sufficient for this task.

Key Results

  • CNNs Outperform RNNs: Across all ad formats, CNN models generally achieved higher performance metrics (F1 score and AUC) compared to RNN models. For instance, the best CNN model showed a significant improvement in AUC over the best RNN model for organic ads (9.35%).
  • Top Performing Representations: For organic ads, colored trajectories with varied line thickness and trajectories with an ad placeholder were top performers. For left-aligned direct display ads, trajectories with an ad placeholder and heatmap representations were strong. For right-aligned ads, trajectories without an ad placeholder yielded the best results.
  • Best Overall Performance: The ResNet50 model, trained on trajectories without an ad placeholder, achieved the best overall AUC (0.739) and F-Measure (0.731) for right-aligned direct display ads.
  • Statistical Significance: Statistical analysis (Friedman's ANOVA and Wilcoxon signed-rank tests) confirmed significant differences in performance between various ad formats and, in some cases, between representations with and without ad placeholders.

Limitations and Future Work

  • RNN Sequence Length: The authors acknowledge that the limited sequence length (50 timesteps) used for RNN training might have constrained their performance compared to CNNs, which processed full visual representations.
  • Desktop Focus: The study was conducted on desktop users, and future work should extend to mobile interactions (touch-based gestures, scrolling).
  • Generalization: While the models show competitive performance, further research could focus on training a general model that predicts attention across all direct display types.
  • Model Architectures: Exploring custom CNN architectures or hybrid RNN-CNN models is suggested.
  • Other Representations: Investigating other color schemas for ad placeholders and more advanced temporal encoding techniques could be beneficial.
  • Broader Applications: The authors emphasize that modeling user attention on SERPs has wide-ranging applications in ranking and UI design, paving the way for future research.

Summary updated

Abstract

Tracking mouse cursor movements can be used to predict user attention on heterogeneous page layouts like SERPs. So far, previous work has relied heavily on handcrafted features, which is a time-consuming approach that often requires domain expertise. We investigate different representations of mouse cursor movements, including time series, heatmaps, and trajectory-based images, to build and contrast both recurrent and convolutional neural networks that can predict user attention to direct displays, such as SERP advertisements. Our models are trained over raw mouse cursor data and achieve competitive performance. We conclude that neural network models should be adopted for downstream tasks involving mouse cursor movements, since they can provide an invaluable implicit feedback signal for re-ranking and evaluation.
BibTeX
@inproceedings{arapakis2020learning,
  author = {Ioannis Arapakis and Luis A. Leiva},
  title = {{Learning Efficient Representations of Mouse Movements to Predict User Attention}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  url = {https://openreview.net/forum?id=-akmk1n7Gc},
}

Loading related content...

Figure