Characterizing the Temporal Dynamics of Information in Visually Guided Predictive Control Using LSTM Recurrent Neural Networks

Kamran Binaee, Anna Starynska, Jeff B. Pelz, Christopher Kanan, Gabriel Jacob Diaz

Abstract

This study investigates the temporal dynamics of predictive control in humans during visually guided actions. Using a virtual reality ball-catching task where visual information is temporarily interrupted, researchers employed Long Short-Term Memory (LSTM) recurrent neural networks to model motor output. The results demonstrate that an integration window as short as 27ms is sufficient to predict gaze and hand movements up to 500ms into the future. These findings suggest that predictive control may rely on discrete mappings of past sensory states to future motor actions rather than complex internal physics simulations.

Introduction

In "visually guided action," humans typically rely on real-time visual feedback to control movements, a process known as online control. However, when visual information is interrupted—such as a ball passing behind an obstacle—the brain must transition to predictive control. The mechanisms underlying this transition remain a subject of debate: does the brain maintain a complex internal model of physics, or does it utilize simpler mappings? This paper addresses the temporal dynamics of such predictions, specifically examining how much past information (the integration window) is required to accurately predict motor output at various future points (the prediction distance).

Methods

The study combined Virtual Reality (VR) experiments with deep learning architectures to analyze human behavior.

Experimental Design

Ten participants (ages 19–30) performed a ball-catching task in an Oculus Rift DK2 environment. During each trial, the ball would "blank" (disappear) for 500ms, forcing participants to rely on predictive control to intercept it with a virtual paddle.

Data and Modeling

The researchers recorded 16-dimensional input vectors at 75Hz, capturing both motor variables (gaze direction, paddle position, and rotation) and optical variables (ball angular position, velocity, depth, size, and looming).

LSTM architecture and feature set used for predictive modeling.

To model this data, a series of LSTM recurrent neural networks were trained. LSTMs were chosen for their ability to capture long-term dependencies in time-series data. The researchers utilized a "subnetwork" approach, where individual networks predicted the motor state at specific future time steps ($\Delta t$). Various integration windows ($I$), ranging from 27ms to 600ms, were tested to determine the minimum necessary historical data for accurate prediction.

Results

The LSTM models demonstrated high accuracy in reproducing human-like interception behavior.

Predictive Accuracy

At a prediction distance of 500ms, the models predicted human gaze behavior within 3° and hand movements within 8.5 cm. Notably, the LSTMs significantly outperformed standard linear regression models. While linear models struggled with trial-by-trial variations, the LSTMs maintained robust performance as the prediction distance increased.

Mean Squared Error (MSE) as a function of prediction distance for various integration windows (I) compared to linear regression.

The "Minimalist" Predictor

A key finding was that increasing the integration window beyond 27ms did not significantly improve prediction accuracy. This suggests that a very brief "snapshot" of the world's state is sufficient for the brain to generate effective predictive motor commands. Even with the shortest integration window, the model accounted for variations in ball trajectories to reproduce human-like interception behavior.

Discussion

The success of the LSTM models, particularly with short integration windows, provides evidence against the necessity of continuous internal physics simulations for predictive control. Instead, the results support a discrete mapping hypothesis: the brain may map specific past sensorimotor states directly to future motor requirements.

Ablation studies further revealed a sensorimotor synergy. While visual features were critical for short-term predictions (increasing error by 31% when removed), longer-term predictions relied on a combination of both visual and kinesthetic information.

Conclusion

This research highlights the efficiency of the human predictive control system. While the current LSTM models do not yet account for biological constraints like visual memory decay or varying perceptual sensitivity, they provide a powerful framework for understanding how the brain bridges gaps in sensory information. Future work will focus on incorporating these biological limitations and exploring dynamic integration windows to better reflect human visual processing.

Summary updated

BibTeX
@inproceedings{binaee2018characterizing,
  author = {Kamran Binaee and Anna Starynska and Jeff B. Pelz and Christopher Kanan and Gabriel Jacob Diaz},
  title = {{Characterizing the Temporal Dynamics of Information in Visually Guided Predictive Control Using LSTM Recurrent Neural Networks}},
  booktitle = {CogSci},
  year = {2018},
  url = {https://openreview.net/forum?id=DmCdiu5f08U},
}
Figure