Policy learning with partial observation and mechanical constraints for multi-person modeling

Keisuke Fujii, Naoya Takeishi, Yoshinobu Kawahara, Kazuya TakedaarXiv 2020 · arXiv · OpenReview

TL;DR

This paper introduces a novel approach to model complex, real-world multi-agent behaviors by incorporating partial observations and mechanical constraints into decentralized policy learning. By using hierarchical variational recurrent neural networks (VRNNs) and a unique binary partial observation mechanism, their method generates more biologically plausible and interpretable agent trajectories, particularly in team sports.

What Problem Does This Paper Address?

The paper tackles the challenge of accurately modeling real-world multi-agent behaviors, especially in complex scenarios like team sports. Existing data-driven models often fall short by assuming agents have complete environmental information and ignore their physical limitations (e.g., body constraints, inertia). This leads to models that are not biologically plausible and are difficult to interpret for behavioral analysis. The authors aim to bridge this gap by developing a model that explicitly accounts for agents' limited observations and their physical constraints, leading to more realistic and interpretable predictions.

Methods

The core of the proposed method is a decentralized policy learning framework that leverages:

  • Hierarchical Variational Recurrent Neural Networks (VRNNs): These models are used for policy learning, allowing them to capture sequential dependencies in agent behavior. The hierarchical structure helps in modeling long-term intent (macro goals) and short-term actions (local policy).
  • Binary Partial Observation Model: This is a key novelty. Instead of assuming full observability, the model learns to represent an agent's observation as a binary vector. This vector indicates the "importance" of other agents' information, allowing the model to visualize and learn which agents an agent is attending to. This is achieved using a Gumbel-softmax reparameterization for differentiable selection.
  • Mechanical Constraints: The model incorporates biomechanical constraints (e.g., minimum-jerk principle for smooth motion) and physical constraints (position, velocity, acceleration relationships) as penalties in the objective function. This encourages the generation of physically realistic movements.
  • Decentralized Macro Goals: To capture long-term intent, the model uses macro goals, which are pretrained and provided as auxiliary information to the local policy. This allows for planning over longer time horizons.

Datasets

The models were trained and evaluated on real-world multi-person motion datasets:

  • Basketball Dataset: Data from the NBA 2015-2016 season.
  • Soccer Dataset: Data from 45 professional soccer league games.

The data was preprocessed to ensure consistency, and specific subsets were used for training, validation, and testing.

Key Insights and Ideas

  • Partial Observation is Crucial for Realism: Ignoring the fact that agents only perceive a subset of the environment leads to unrealistic behaviors. The binary observation model effectively captures this limitation and provides interpretability by showing what information agents prioritize.
  • Mechanical Constraints Enhance Plausibility: Explicitly modeling physical constraints like inertia and the relationship between position, velocity, and acceleration leads to smoother and more biologically realistic trajectories, reducing constraint violations.
  • Decentralization is Key for Team Sports: Modeling agents as decentralized entities is essential for capturing the dynamics of team sports, where individual agents make decisions based on their limited perception and local goals.
  • Interpretability Through Observation Coefficients: The learned binary observation coefficients offer a direct way to understand which agents an agent is attending to, providing valuable insights into their decision-making process.

Key Results

  • Reduced Constraint Violations: The proposed method (VRNN-macro-Bi-Mech) demonstrated significantly smaller mechanical constraint losses (Lpos, Lacc, Ljrk) compared to baseline VRNN models on both basketball and soccer datasets.
  • Physically Plausible Trajectories: The models generated more physically plausible and smoother trajectories, as evidenced by lower constraint losses, particularly for acceleration smoothing (Ljrk).
  • Competitive Prediction Performance: While not the primary focus, the proposed models achieved comparable long-term trajectory prediction performance to existing methods, especially when mechanical constraints were incorporated.
  • Effective Partial Observation Modeling: The binary observation model was shown to effectively capture agent observations, visualizing which players were attended to, even in complex scenarios.

Limitations and Future Work

  • Soft Constraints: The paper uses soft constraints (penalties) rather than hard constraints due to data resolution limitations. This means the constraints are not strictly enforced.
  • Trade-offs in Constraints: There's a noted trade-off between smoothing acceleration (Ljrk) and improving acceleration prediction performance.
  • No Ground Truth for Observation: Evaluating the observation model is challenging due to the lack of ground truth for what agents actually observe. The authors validate it based on domain knowledge and qualitative analysis.
  • Future Work: The authors suggest incorporating other physiological constraints like visuomotor delays and body loads to further enhance the understanding of smooth multi-agent behaviors in complex environments.

Summary updated

Abstract

Extracting the rules of real-world biological multi-agent behaviors is a current challenge in various scientific and engineering fields. Biological agents generally have limited observation and mechanical constraints; however, most of the conventional data-driven models ignore such assumptions, resulting in lack of biological plausibility and model interpretability for behavioral analyses in biological and cognitive science. Here we propose sequential generative models with partial observation and mechanical constraints, which can visualize whose information the agents utilize and can generate biologically plausible actions. We formulate this as a decentralized multi-agent imitation learning problem, leveraging binary partial observation models with a Gumbel-Softmax reparameterization and policy models based on hierarchical variational recurrent neural networks with physical and biomechanical constraints. We investigate the empirical performances using real-world multi-person motion datasets from basketball and soccer games.
BibTeX
@misc{fujii2020policy,
  author = {Keisuke Fujii and Naoya Takeishi and Yoshinobu Kawahara and Kazuya Takeda},
  title = {{Policy learning with partial observation and mechanical constraints for multi-person modeling}},
  year = {2020},
  eprint = {2007.03155},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.03155},
}
Figure