Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors

Julian Steil, Philipp Müller, Yusuke Sugano, Andreas Bulling

Abstract

Visual attention is highly fragmented during mobile interactions, but the erratic nature of attention shifts currently limits attentive user interfaces to adapting after the fact, i.e. after shifts have already happened. We instead study attention forecasting—the challenging task of predicting users' gaze behaviour (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users' visual scene. We demonstrate that our method can forecast bidirectional attention shifts and predict whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.

Introduction

Our attention during mobile interactions is notoriously fragmented as we constantly glance back and forth between our screens and the surrounding world. Currently, "attentive user interfaces" are purely reactive; they only adapt after a user has already looked away—for instance, pausing a video only after a scene has been missed.

To move toward proactive interfaces that can alert users to hazards before they look down or pre-load content before they glance at their device, we must be able to forecast these shifts in the near future. Attention shifts are driven by a complex interplay of external "bottom-up" distractors (e.g., a passing car) and internal "top-down" goals (e.g., checking a notification). This study explores the feasibility of predicting these erratic shifts using multimodal sensor data.

The proposed system uses egocentric vision and phone-integrated sensors to forecast bidirectional attention shifts and primary attentional focus.

Methodology

The researchers developed a multimodal machine learning approach to forecast overt visual attention (gaze). The system processes three primary data streams:

  1. Egocentric Vision: A head-mounted camera extracts semantic information, including object detection (counting people or vehicles), face detection, and semantic scene segmentation (identifying environment types like offices or libraries).
  2. Device Usage: Smartphone-integrated sensors (accelerometer, gyroscope) and application logs (active apps, touch events, screen status) provide context on user-device interaction.
  3. User State: Head movement data from a head-worn IMU and, in specific configurations, historical gaze data from an eye tracker are incorporated.

A Random Forest classifier was employed to process these features, evaluating performance across different "target windows"—predicting 1, 5, and 10 seconds into the future.

The forecasting task distinguishes between discrete attention shifts (A) and the sustained primary attentional focus (B) over specific target windows.

Dataset and Evaluation

The study introduced a novel dataset of everyday mobile interactions involving 20 participants roaming a university campus freely for 4.5 hours each, totaling over 90 hours of recording. Unlike scripted studies, participants engaged in natural activities like visiting cafes and libraries while performing chat-based tasks. The data was meticulously annotated for environment type, locomotion mode (sitting, standing, walking), and every individual attention shift.

Key Results and Insights

The evaluation focused on three main forecasting tasks using the weighted F1 score:

  • Shifts to Environment: Achieved an F1 score of 0.76 (during active tasks) using a 1-second prediction window. These shifts are primarily driven by the visual scene (bottom-up attractors), making egocentric camera data the most valuable feature.
  • Shifts to Mobile Device: Achieved an F1 score of 0.83 using a 10-second window. These shifts are driven by device state and internal goals; phone-based features alone were highly effective here, outperforming camera data.
  • Primary Attentional Focus: Predicting the user's main focus in the next 5 seconds reached an F1 score of >0.70, increasing to >0.80 when past gaze data was included.

Performance varied by environment, peaking in structured areas like corridors and libraries, while busy streets proved more challenging due to unpredictable visual distractors. Furthermore, predicting shifts to the phone worked better over longer windows (10s) because the physical act of picking up or unlocking a phone provides a significant lead-in signal.

Discussion and Future Work

While this proof-of-concept is successful, several hurdles remain for real-world deployment. The current hardware setup is bulky, requiring a backpack and laptop; future iterations must integrate these sensors into lightweight smart glasses. Additionally, the model currently ignores auditory triggers, which are major drivers of attention. Future research should also address the "where" of attention shifts—not just the "when"—and tackle the privacy concerns associated with using egocentric cameras in public spaces through on-device processing.

Summary updated

Abstract

Visual attention is highly fragmented during mobile interactions, but the erratic nature of attention shifts currently limits attentive user interfaces to adapting after the fact, i.e. after shifts have already happened. We instead study attention forecasting -- the challenging task of predicting users' gaze behaviour (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users' visual scene. We demonstrate that our method can forecast bidirectional attention shifts and predict whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.
BibTeX
@inproceedings{steil2018forecasting,
  author = {Julian Steil and Philipp Müller and Yusuke Sugano and Andreas Bulling},
  title = {{Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors}},
  booktitle = {MobileHCI},
  year = {2018},
  url = {https://openreview.net/forum?id=NJCX6vU3wrq},
}
Figure