SHA-SCP: A UI Element Spatial Hierarchy Aware Smartphone User Click Behavior Prediction Method

Ling Chen, Qian Chen, Yiyi Peng, Kai Qian, Hongyu Shi, Xiaofan ZhangIEEE Trans. Hum. Mach. Syst 2025 · DOI

TL;DR

This paper introduces SHA-SCP, a novel method for predicting the next user click on a smartphone. By considering the spatial hierarchy of UI elements and using attention mechanisms at both element and group levels, SHA-SCP significantly improves prediction accuracy, offering a more nuanced understanding of user interaction. This advancement is crucial for enhancing user experience through personalized app features and faster response times.

What Problem Does This Paper Address?

Predicting a user's next click on their smartphone is vital for improving user experience, enabling personalized recommendations, and speeding up app interactions. Existing methods often fall into two categories: coarse-grained (predicting the next app) and fine-grained (predicting the next UI element click). While coarse-grained methods overlook detailed interactions within an app, fine-grained methods struggle with the sheer complexity and dynamic nature of UIs, making it difficult to capture information across different scales. Specifically, previous fine-grained approaches either focus on a limited set of elements or struggle to precisely identify and model all elements due to their dynamic positions. This paper aims to bridge this gap by proposing a method that effectively captures both local element details and broader structural information within a UI.

Methods

The core of SHA-SCP lies in its "UI element spatial hierarchy awareness." The method consists of five modules:

  • Element Embedding: This module creates an element sequence from the UI's view hierarchy and generates initial embeddings for each element based on its attributes (text, type, position).
  • Element Spatial Hierarchy Awareness: This is the key novelty. It clusters UI elements based on their spatial positions into "element groups." This allows the model to perceive the UI at a higher, group level, capturing larger-scale information.
  • Hierarchical Element Encoding: This module uses attention mechanisms to process information at both the individual element level and the element group level. This dual-level perception allows SHA-SCP to capture information at different scales simultaneously.
  • Click Sequence Modeling: A Transformer model is employed to model the historical sequence of clicks, capturing temporal dependencies and generating a latent representation of the next predicted click.
  • Next User Click Behavior Prediction: This final module uses the latent representation, the current UI's elements, and contextual information to predict the most likely next clicked element.

The spatial hierarchy awareness is achieved by first clustering elements using DBSCAN and then aggregating their features to form element group embeddings. These group embeddings are then processed alongside individual element embeddings through attention mechanisms.

Datasets

The paper introduces a newly collected fine-grained smartphone usage behavior dataset. This dataset comprises:

  • 3,664,325 clicks
  • From 100 users
  • Across 1,087 apps
  • Collected over 30 days.

This dataset captures detailed UI element clicks, including the UI, clicked element, and contextual information (time, app). The data was collected using a custom app on Android and HarmonyOS devices.

Key Insights and Ideas

  • Spatial Hierarchy Matters: The most significant insight is that simply looking at individual UI elements is insufficient. Grouping elements based on their spatial proximity reveals a hierarchical structure that is crucial for understanding user intent and predicting clicks.
  • Multi-Scale Perception: By attending to both individual elements and element groups, SHA-SCP can effectively capture information at different granularities, from fine details to broader structural patterns within a UI.
  • Attention is Key: The use of attention mechanisms at both the element and group levels allows the model to dynamically weigh the importance of different UI components and their relationships, leading to more accurate predictions.
  • Beyond Individual Elements: The success of SHA-SCP highlights that user interaction is not just a sequence of individual clicks but is influenced by the overall layout and organization of the UI.

Key Results

  • SHA-SCP achieved significant improvements over state-of-the-art baselines:
  • Top-1 Accuracy: 35.34% (vs. 24.82% for Google Model)
    • Top-3 Accuracy: 56.84% (vs. 45.50% for Google Model)
    • Top-5 Accuracy: 68.27% (vs. 57.85% for Google Model)
  • These improvements translate to an average gain of 10.52% in Top-1 Accuracy, 11.34% in Top-3 Accuracy, and 10.42% in Top-5 Accuracy compared to the best baseline.
  • Experiments with simplified models showed that removing either element-level or group-level attention significantly degraded performance, underscoring the importance of both.
  • Feature analysis indicated that text, position, and type are the most important features for prediction.

Limitations and Future Work

  • Element Grouping: The current method uses a distance-based clustering for element groups, which might not adapt well to all user behaviors or UI types. Future work will explore other parameterized methods.
  • Efficiency: SHA-SCP is computationally intensive. The authors plan to investigate sparse attention mechanisms to improve efficiency without sacrificing accuracy.
  • Contextual Information: The current dataset is limited in contextual information (e.g., location). Future work aims to collect datasets with richer context.
  • Broader Interactions: The model currently focuses only on click prediction. Future research will explore predicting other types of user interactions.
  • Model Compression: The model is too large for direct deployment on smartphones. Efforts will be made to compress the model for on-device execution.

Summary updated

BibTeX
@article{chen2025sha,
  author = {Ling Chen and Qian Chen and Yiyi Peng and Kai Qian and Hongyu Shi and Xiaofan Zhang},
  title = {{SHA-SCP: A UI Element Spatial Hierarchy Aware Smartphone User Click Behavior Prediction Method}},
  journal = {IEEE Trans. Hum. Mach. Syst.},
  year = {2025},
  doi = {10.1109/thms.2025.3601578},
}
Figure