Linear Embodied Saliency: a Model of Full-Body Kinematics-based Visual Attention

William W. Abbott, J. Alex Harston, A. Aldo FaisalbioRxiv 2020 · DOI

Abstract

Gaze behaviour and motor actions are fundamentally interlinked in both a spatial and temporal manner. However, the vast majority of gaze behaviour research has focused to date on reductionist head-fixed screen viewing experiments and ignored the motor aspect of visuomotor behaviour, thereby neglecting a critical component of the perception-action loop. We address this with an experimental design to capture, rather than constrain, the full range of simultaneous gaze and motor behaviour in a range of natural daily life tasks. Through building autoregressive models and applying these to our novel datasets we find that beyond simple static regions of interest, we can predict visual attention shifts from freely-moving first person body kinematics, through explaining gaze dynamics in the context of body dynamics, on the timescale of freely moving interactive behaviour in individuals, expanding our understanding of natural visuomotor behaviour.

Introduction

For decades, vision science has sought to understand the drivers of human visual attention. Traditional theories typically categorize gaze behavior into "bottom-up" saliency—where low-level image features like contrast and color draw the eye—and "top-down" attention—where high-level goals dictate where we look. However, these approaches often overlook the "embodied" nature of vision. In natural environments, humans are active agents; our eyes, head, and limbs move in a tightly coordinated perception-action loop. Most existing research relies on "disembodied" setups where subjects' heads are fixed while viewing 2D screens, failing to capture the rich interplay between full-body kinematics and gaze.

Methods and Data Collection

To bridge this gap, we developed an experimental framework to capture "ethomic" data—high-resolution behavioral data in natural habitats. Participants were outfitted with a suite of wearable, markerless sensors to record their movements and gaze simultaneously.

Experimental setup showing the motion capture suit, eye-tracking glasses, and 3D skeleton reconstruction.
  • Gaze Tracking: SMI Eye Tracking Glasses (30Hz) recorded the point of regard.
  • Body Kinematics: An IGS-180 motion capture suit measured 51 degrees of freedom (DOF) across the body.
  • Hand Tracking: CyberGloves captured 40 DOF in the hands.

We introduced a novel dataset consisting of three natural scenarios: "Breakfast Time" (e.g., making tea), "Evening Chores" (e.g., making the bed), and "Navigation/Socializing." This dataset provides continuous, time-locked streams of 3D body pose and gaze from a first-person perspective.

Modeling Embodied Saliency

We modeled the relationship between movement and vision using Vector AutoRegressive models with eXogenous inputs (VARX). These models mapped the 51 body DOFs to the 2 DOFs of gaze (pitch and yaw). We compared three frameworks: Gaze Autoregression (Auto), Exogenous Regression (Exo) based only on body angles, and a hybrid AutoExo model.

Key Insights and Results

Our analysis revealed that the body acts as a powerful "pointer" for visual attention. Body movements alone can explain up to 40% of eye-movement variance in natural tasks.

Visualization of time-locked gaze and body kinematics data used for the linear regression models.

Predictive Performance

The linear models achieved significant predictive power, with an $R^2$ of approximately 0.35 for horizontal gaze and 0.22 for vertical gaze. Notably, these embodied models outperformed traditional saliency benchmarks and the common "central fixation bias" by 200-300%.

Temporal Dynamics

The "AutoExo" model proved most effective at intermediate timescales (200ms to 2 seconds). At very short intervals (<200ms), gaze is highly predictable from its own trajectory. Furthermore, we found that joint sets including head dynamics (head, spine, and hips) were the strongest predictors of gaze, significantly outperforming models based solely on limb movements.

Discussion and Future Work

While our linear models demonstrate that a substantial portion of gaze behavior is rooted in body kinematics, approximately 60% of the variance remains unexplained. This suggests that while physical posture provides a strong baseline, complex cognitive factors and non-linear interactions also play critical roles.

Future research will focus on non-linear architectures like deep learning to capture more complex behaviors and exploring "proactive planning"—whether current gaze can predict future motor actions. By treating the observer as an embodied agent, this work moves beyond static saliency toward a more holistic understanding of the perception-action loop in daily life.

Summary updated

Abstract

Abstract Gaze behaviour and motor actions are fundamentally interlinked in both a spatial and temporal manner. However, the vast majority of gaze behaviour research has focused to date on reductionist head-fixed screen viewing experiments and ignored the motor aspect of visuomotor behaviour, thereby neglecting a critical component of the perception-action loop. We address this with an experimental design to capture, rather than constrain, the full range of simultaneous gaze and motor behaviour in a range of natural daily life tasks. Through building autoregressive models and applying these to our novel datasets we find that beyond simple static regions of interest, we can predict visual attention shifts from freely-moving first person body kinematics, through explaining gaze dynamics in the context of body dynamics, on the timescale of freely moving interactive behaviour in individuals, expanding our understanding of natural visuomotor behaviour.
BibTeX
@misc{abbott2020linear,
  author = {William W. Abbott and J. Alex Harston and A. Aldo Faisal},
  title = {{Linear Embodied Saliency: a Model of Full-Body Kinematics-based Visual Attention}},
  year = {2020},
  doi = {10.1101/2020.02.08.938514},
  howpublished = {bioRxiv},
}
Figure