A computational framework for understanding embodied plans

Aalap D. Shah, Basil Saeed, Joshua B. Tenenbaum, Tobias Gerstenberg, Hyowon Gweon, Ilker YildirimarXiv 2026 · arXiv

TL;DR

Researchers have introduced GenEA (Generative Embodied Agent), a computational framework that explains how humans reason about the physical plans of others. By combining robotics-style planning with simulation-based intuitive physics, GenEA accurately predicts human judgments of goals, actions, and task difficulty across complex manipulation and navigation scenarios.

What Problem Does This Paper Address?

When you watch a child reach for a block or a person jump over a puddle, you don't just see movement; you see a plan. You can infer their goal, judge if the action is effortful, and even predict if they might fall.

Historically, cognitive science has modeled this "social reasoning" using simplified, point-like agents in 2D grid worlds. However, real-world reasoning is embodied: it depends on the biomechanics of the body (e.g., using one hand vs. two) and the physics of the environment (e.g., gravity and object stability). The challenge has been formalizing this rich, multi-step reasoning in a way that aligns with human intuition.

Methods

The authors present GenEA, which represents an agent's plan through a multilevel state-space. It extends the standard Task-and-Motion Planning (TAMP) framework from robotics by adding a layer of intuitive physics.

The model operates across three levels:

  1. Symbolic Level ($f^{(S)}$): Defines high-level actions like PICK, FIX, STEP, or JUMP.
  2. Kinematics Level ($f^{(K)}$): Translates symbolic actions into noisy 3D motion trajectories for the agent's joints.
  3. Dynamics Level ($f^{(D)}$): Uses a physics engine (PhysX or MuJoCo) to simulate how actions affect the world, accounting for gravity, mass, and friction.
GenEA's multilevel state-space and planning architecture.

GenEA treats planning as a generative process. It calculates a joint probability distribution over goals and plans, where the probability of a plan is inversely related to its cost (specifically, the total kinetic energy required). To mimic human decision-making, the model uses a choice rule based on the relative probability of different options.

Datasets

The framework was evaluated against human data ($N=60$ per experiment) across three distinct preregistered experiments:

  • Experiment 1 (Goal & Action Inference): Participants inferred an agent's intended block structure or their first action based on initial/final states.
  • Experiment 2 (Embodiment Strategies): Participants judged whether a block-stacking task required one hand or two.
  • Experiment 3 (Navigation Difficulty): Participants rated the difficulty of an agent traversing an obstacle course with varying platform shapes and materials (wood vs. stone).

The model was compared against several baselines: a standard TAMP model (no physics), task-specific heuristics, and a state-of-the-art vision-language model (ChatGPT v5.0).

Key Insights and Ideas

  • Physics is Essential for Social Cognition: The "aha moment" is that standard robotics planning (TAMP) isn't enough to explain human reasoning. Only when the model accounts for dynamics (like the risk of a tower toppling or a platform sliding) does it align with human judgment.
  • Embodied Cost Functions: Humans seem to evaluate plans based on physical efficiency. GenEA’s use of kinetic energy as a cost metric outperformed simpler metrics like "plan length."
  • Predicting the "Process," not just the "Result": Remarkably, GenEA predicted human response times. Trials where the model found two options nearly equal in cost (high ambivalence) corresponded to trials where humans took longer to decide, suggesting the model captures the underlying cognitive "evidence accumulation" process.
  • Material Matters: Humans (and GenEA) correctly perceive that wooden platforms are "harder" to navigate than stone ones because they are lighter and less stable. Models without physics are blind to this distinction.

Key Results

GenEA consistently outperformed all alternative models across all metrics:

  • High Correlation with Humans: GenEA achieved Pearson correlations of $r = 0.88$ for goal/action inference, $r = 0.93$ for embodiment strategies, and $r = 0.91$ for navigation difficulty.
  • Superiority over VLMs: Despite its massive scale, the vision-language model (ChatGPT v5.0) struggled to match human physical intuitions, falling significantly behind GenEA in predictive accuracy.
  • Robustness: The results held steady across different cost functions and temperature parameters, proving the framework's core architecture is what drives its success.
GenEA's performance in Experiment 1 and 2 compared to human judgments.

Limitations and Future Work

  • Structured Input: Currently, GenEA requires a structured state-space (knowing where objects are and their properties). The authors suggest future work should integrate perception systems to make the model "image-computable" directly from raw pixels.
  • Broadening the Scope: The framework could be extended to reason about higher-order agent properties like skill, motivation, and persistence.
  • Algorithmic Implementation: While GenEA provides a "computational-level" account (what the mind is calculating), exactly how the brain approximates these complex simulations remains an open question for future algorithmic research.

Summary updated

Abstract

Reasoning about people's plans and goals based on their actions is fundamental to human social interaction and essential for cooperative, embodied artificial intelligence. Formalizing this capacity in naturalistic, embodied settings has been a longstanding challenge. Here, we present GenEA, a novel agent representation capable of reasoning over multi-step, long-horizon embodied plans. At the heart of GenEA lies a multilevel state-space that couples the coarse biomechanics of an articulated body and its interactions with nearby objects by extending the task-and-motion planning framework from robotics with simulation-based intuitive physics. By embedding this state-space in a generative model, GenEA yields a joint probability distribution over goals and embodied plans. Across several experiments spanning the qualitatively distinct domains of object manipulation and navigation, featuring diverse tasks and multiple behavioral measures, we show that GenEA, through simple transformations of this joint distribution, recapitulates how humans reason about embodied plans. Crucially, GenEA consistently outperforms strong alternative models, including plausible heuristics, standard task-and-motion planning, and a state-of-the-art vision-language model. GenEA thereby offers a new perspective on the architecture of human reasoning and social cognition, suggesting that embodied artificial intelligence systems seeking to seamlessly interact with humans should adopt GenEA-like generative models of action planning.
BibTeX
@misc{shah2026computational,
  author = {Aalap D. Shah and Basil Saeed and Joshua B. Tenenbaum and Tobias Gerstenberg and Hyowon Gweon and Ilker Yildirim},
  title = {{A computational framework for understanding embodied plans}},
  year = {2026},
  eprint = {2611.13669},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13669},
}
Figure