Abstract
Despite increasing attention paid to the need for fast, scalable methods to analyze next-generation neuroscience data, comparatively little attention has been paid to the development of similar methods for behavioral analysis. Just as the volume and complexity of brain data have grown, behavioral paradigms in systems neuroscience have likewise become more naturalistic and less constrained, necessitating an increase in the flexibility and scalability of the models used to study them. In particular, key assumptions made in the analysis of typical decision paradigms — optimality; analytic tractability; discrete, low-dimensional action spaces — may be untenable in richer tasks. Here, using the case of a two-player, real-time, continuous strategic game as an example, we show how the use of modern machine learning methods allows us to relax each of these assumptions. Following an inverse reinforcement learning approach, we are able to succinctly characterize the joint distribution over players' actions via a generative model that allows us to simulate realistic game play. We compare simulated play from a number of generative time series models and show that ours successfully resists mode collapse while generating trajectories with the rich variability of real behavior. Together, these methods offer a rich class of models for the analysis of continuous action tasks at the single-trial level.
Introduction
In systems neuroscience, behavior is traditionally studied using highly constrained, discrete tasks, such as choosing between two buttons. However, natural behavior is continuous, high-dimensional, and often involves complex strategic interactions between multiple agents. Traditional analysis methods, which rely on averaging data across trials aligned to specific events, struggle with these unconstrained environments where clear "events" are absent.
Furthermore, standard generative models often suffer from mode collapse. When an agent faces multiple valid strategic options (e.g., moving left or right to avoid an opponent), these models tend to average the possibilities, resulting in a single, unrealistic trajectory. This research introduces a framework to capture the multi-modal richness of strategic behavior by modeling latent intentions.
Methods: Decoupling Intention from Execution
The proposed hierarchical generative model separates high-level "intentions" from low-level "execution." This decoupling is achieved through four primary components:
- The Goal Model: Instead of modeling raw motor commands, the model infers a latent "goal" ($g_t$)—the onscreen position an agent intends to reach. The evolution of these goals is modeled as a Markov process.
- GMM Value Function: To represent multiple potential strategies, the value function is approximated using a Gaussian Mixture Model (GMM). A neural network outputs the parameters (weights, means, and variances) of this mixture based on the current game state. This allows the model to represent probability distributions with multiple peaks, preventing mode collapse.
- The Control Model: A Proportional-Integral-Derivative (PID) controller translates the high-level goal into movement, mimicking biological motor control.
- Inference: The model is trained using Variational Bayes, specifically a "black box" approach to maximize the Evidence Lower Bound (ELBO), learning strategies directly from trajectory data.
Experimental Setup: The Penalty Shot Game
The model was evaluated using a two-player "Penalty Shot" video game played by rhesus macaques. In this task, one player controls a ball moving in 2D, while the other controls a goalie moving vertically. The objective for the ball is to cross the goal line, while the goalie attempts to intercept it. The dataset comprised approximately 7,000 bouts across 10 sessions, recorded as multivariate time series of positions and velocities.
Key Results and Insights
The GMM-based model successfully reproduced the diverse "fan" of trajectories observed in real animal behavior. Unlike baseline models such as the Deep Latent Gaussian Model (DLGM), which collapsed into a few stereotypical paths, the proposed framework maintained the rich variability of strategic play.
By analyzing the latent goals, the researchers could visualize a player's "intent" before physical movement occurred. The model also demonstrated sensitivity to initial states; varying the starting "goal" led to different simulated strategic outcomes, such as feints and counter-moves. For neuroscientists, the model provides mathematically grounded variables—such as "momentary surprise" and "goal mixture entropy"—that can be correlated with neural activity to identify the biological substrates of strategic decision-making.
Limitations and Future Work
Currently, the model assumes trials are independent and identically distributed (I.I.D.), ignoring potential learning effects across a session. Future iterations could incorporate more biologically realistic motor noise or scale to more complex environments with additional agents, potentially utilizing structured temporal models like Hierarchical Hidden Markov Models.
Summary updated