Abstract
This study investigates the temporal dynamics of predictive control in humans during visually guided actions. Using a virtual reality ball-catching task where visual information is temporarily interrupted, researchers employed Long Short-Term Memory (LSTM) recurrent neural networks to model motor output. The results demonstrate that an integration window as short as 27ms is sufficient to predict gaze and hand movements up to 500ms into the future. These findings suggest that predictive control may rely on discrete mappings of past sensory states to future motor actions rather than complex internal physics simulations.
Introduction
In "visually guided action," humans typically rely on real-time visual feedback to control movements, a process known as online control. However, when visual information is interrupted—such as a ball passing behind an obstacle—the brain must transition to predictive control. The mechanisms underlying this transition remain a subject of debate: does the brain maintain a complex internal model of physics, or does it utilize simpler mappings? This paper addresses the temporal dynamics of such predictions, specifically examining how much past information (the integration window) is required to accurately predict motor output at various future points (the prediction distance).
Methods
The study combined Virtual Reality (VR) experiments with deep learning architectures to analyze human behavior.
Experimental Design
Ten participants (ages 19–30) performed a ball-catching task in an Oculus Rift DK2 environment. During each trial, the ball would "blank" (disappear) for 500ms, forcing participants to rely on predictive control to intercept it with a virtual paddle.
Data and Modeling
The researchers recorded 16-dimensional input vectors at 75Hz, capturing both motor variables (gaze direction, paddle position, and rotation) and optical variables (ball angular position, velocity, depth, size, and looming).
To model this data, a series of LSTM recurrent neural networks were trained. LSTMs were chosen for their ability to capture long-term dependencies in time-series data. The researchers utilized a "subnetwork" approach, where individual networks predicted the motor state at specific future time steps ($\Delta t$). Various integration windows ($I$), ranging from 27ms to 600ms, were tested to determine the minimum necessary historical data for accurate prediction.
Results
The LSTM models demonstrated high accuracy in reproducing human-like interception behavior.
Predictive Accuracy
At a prediction distance of 500ms, the models predicted human gaze behavior within 3° and hand movements within 8.5 cm. Notably, the LSTMs significantly outperformed standard linear regression models. While linear models struggled with trial-by-trial variations, the LSTMs maintained robust performance as the prediction distance increased.
The "Minimalist" Predictor
A key finding was that increasing the integration window beyond 27ms did not significantly improve prediction accuracy. This suggests that a very brief "snapshot" of the world's state is sufficient for the brain to generate effective predictive motor commands. Even with the shortest integration window, the model accounted for variations in ball trajectories to reproduce human-like interception behavior.
Discussion
The success of the LSTM models, particularly with short integration windows, provides evidence against the necessity of continuous internal physics simulations for predictive control. Instead, the results support a discrete mapping hypothesis: the brain may map specific past sensorimotor states directly to future motor requirements.
Ablation studies further revealed a sensorimotor synergy. While visual features were critical for short-term predictions (increasing error by 31% when removed), longer-term predictions relied on a combination of both visual and kinesthetic information.
Conclusion
This research highlights the efficiency of the human predictive control system. While the current LSTM models do not yet account for biological constraints like visual memory decay or varying perceptual sensitivity, they provide a powerful framework for understanding how the brain bridges gaps in sensory information. Future work will focus on incorporating these biological limitations and exploring dynamic integration windows to better reflect human visual processing.
Summary updated