Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling

Keisuke Fujii, Naoya Takeishi, Yoshinobu Kawahara, Kazuya Takeda

This academic paper, recently updated in late 2023, presents a novel framework for modeling how multiple humans (like athletes) interact in complex environments. By moving away from "omniscient" centralized models and incorporating the laws of physics and biology, the authors provide a more interpretable and realistic way to simulate multi-agent behavior.

TL;DR

The researchers propose a decentralized imitation-learning model that mimics how biological agents actually behave: with limited vision and physical bodies. By combining hierarchical Variational Recurrent Neural Networks (VRNNs) with biomechanical penalties and a "binary" attention mechanism, they generate sports trajectories that are not only accurate but also physically plausible and interpretable.

What Problem Does This Paper Address?

Most current data-driven models for multi-agent systems (like predicting player movement in basketball or soccer) make two unrealistic assumptions:

  1. Centralization: They assume a single "brain" controls all agents or that every agent knows exactly what everyone else is doing at all times.
  2. Lack of Physics: They treat agents as points in space that can change direction or speed instantly, ignoring the mechanical constraints of the human body (like inertia and joint limits).

This leads to "black box" models that produce jittery, biologically impossible movements and offer no insight into why a player made a specific decision. The authors aim to bridge the gap between high-performance machine learning and biological plausibility.

Methods

The authors formulate the problem as a Decentralized Partially Observable Markov Decision Process (POMDP). The key technical innovations include:

  • Binary Partial Observation Model: Instead of "soft attention" (which gives a tiny bit of weight to everyone), they use a Gumbel-softmax reparameterization to create a binary vector. This explicitly shows which specific agents a player is "observing" at any given moment, making the model's "vision" interpretable.
  • Mechanical Constraints as Penalties: Rather than using "hard" constraints (which fail on noisy real-world data), they introduce "soft" penalties into the loss function. These include a minimum-jerk principle (to ensure smooth movement) and physical consistency penalties (ensuring that predicted velocity actually matches the change in position).
  • Hierarchical VRNNs: They use a Variational Recurrent Neural Network to handle the inherent uncertainty and multi-modal nature of human behavior (e.g., a player might go left or right to avoid a screen).
  • Macro Goals: The model incorporates "macro goals"—long-term intents like "run to the corner"—to guide the short-term path planning.

Datasets

The model was validated using two high-stakes, real-world datasets:

  1. NBA Basketball Data (2015-2016 Season): Tracking data for players and the ball, preprocessed to focus on defensive team policies.
  2. Professional Soccer Data: Trajectories from 45 professional league games, involving a larger playing area and more agents than the basketball set.

Key Insights and Ideas

  • Physics as a Regularizer: The authors found that learning acceleration is notoriously difficult because of data noise. By adding biomechanical penalties, they forced the model to produce smoother, more "human-like" trajectories without sacrificing prediction accuracy.
  • Visualizing Cognition: Because the observation model is binary, researchers can literally see who a defender is watching. For example, the model showed that expert defenders often balance their attention between the ball and their direct opponent, even if other players are closer.
  • Counterfactual Simulation: A major "aha moment" is the ability to perform counterfactuals. By manually overriding the observation vector (e.g., "What if this defender only looked at the ball?"), the model can simulate how a "beginner" might react compared to a pro.

Key Results

  • Reduced Violations: The "VRNN-Mech" variant significantly reduced violations of physical laws (position and acceleration consistency) compared to standard VRNNs.
  • Competitive Accuracy: Despite the added constraints, the model maintained $L_2$ prediction errors (distance from ground truth) comparable to state-of-the-art unconstrained models (around 0.7m for basketball position).
  • Qualitative Smoothness: Qualitative analysis showed that the proposed model avoided the erratic "teleportation-like" movements often seen in pure RNN-based predictors.

Limitations and Future Work

The authors note a few areas for improvement:

  • Macro Goal Coordination: In the decentralized setting, macro goals (long-term intent) didn't significantly boost raw prediction accuracy, likely because the agents weren't "sharing" their long-term plans with each other.
  • Validation of "Attention": Since there is no ground truth for what a player was actually looking at during a game, the interpretability of the observation model relies on domain knowledge rather than direct measurement.
  • Future Physiological Factors: The authors suggest that future models should incorporate visuomotor delays (the time it takes for a human to react to a visual stimulus) and body loads to further increase realism.

Summary updated

Abstract

Extracting the rules of real-world multi-agent behaviors is a current challenge in various scientific and engineering fields. Biological agents independently have limited observation and mechanical constraints; however, most of the conventional data-driven models ignore such assumptions, resulting in lack of biological plausibility and model interpretability for behavioral analyses. Here we propose sequential generative models with partial observation and mechanical constraints in a decentralized manner, which can model agents' cognition and body dynamics, and predict biologically plausible behaviors. We formulate this as a decentralized multi-agent imitation-learning problem, leveraging binary partial observation and decentralized policy models based on hierarchical variational recurrent neural networks with physical and biomechanical penalties. Using real-world basketball and soccer datasets, we show the effectiveness of our method in terms of the constraint violations, long-term trajectory prediction, and partial observation. Our approach can be used as a multi-agent simulator to generate realistic trajectories using real-world data.
BibTeX
@article{fujii2024decentralized,
  author = {Keisuke Fujii and Naoya Takeishi and Yoshinobu Kawahara and Kazuya Takeda},
  title = {{Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling}},
  journal = {Neural Networks},
  year = {2024},
  doi = {10.1016/j.neunet.2023.11.068},
}
Figure