PHYLOMAN: Generative Behavior Control via Fusing LLM Planning and Physics-Based Control

Jusheng Zhang, Jinzhou Tang, Sidi Liu, Jian Wang, Keze WangCVPR 2026 Findings 2026 · OpenReview

TL;DR

PHYLOMAN bridges the gap between high-level AI planning and low-level robotic execution by combining Large Language Models (LLMs) with physics-informed motion diffusion. By introducing a new task called Generative Behavior Control (GBC) and a massive dataset (GBC-100K), this framework generates purposeful, physically plausible humanoid behaviors over timelines up to 10× longer than previous state-of-the-art methods.

What Problem Does This Paper Address?

Current human motion generation research is highly fragmented. On one hand, deep generative models (like motion diffusion) excel at synthesizing short, realistic movements but lack long-term purpose or semantic goal-orientation. On the other hand, high-level task planners can map out complex actions but are "disembodied," completely detached from physical constraints like joint limits, balance, and gravity. Classic Task and Motion Planning (TAMP) frameworks try to bridge this gap but are often too rigid and deterministic to capture the fluid diversity of human behavior.

The authors formalize this missing middle ground as Generative Behavior Control (GBC): the task of generating long-horizon action sequences that are simultaneously goal-oriented, physically plausible, and semantically coherent.

Methods

PHYLOMAN solves the GBC task using a hierarchical architecture inspired by TAMP principles:

  1. High-Level Behavior Planning: An LLM acts as the task planner. Given an abstract prompt (e.g., "repairing a car in a garage"), it decomposes the goal into a structured BehaviorScript consisting of discrete keyframe postures (PoseScripts) and transitional movements (MotionScripts).
  2. Multi-segment Parallel Motion Diffusion Model (MP-MDM): Instead of generating long sequences sequentially—which is computationally slow and prone to drifting—MP-MDM co-designs transitions and target poses simultaneously. It uses a Variational Autoencoder (VAE) to encode keyframe poses and a diffusion model to interpolate transitions in parallel across the timeline ("parallel-in-time"), drastically boosting efficiency.
  3. Low-Level Motion Control: The generated kinematic motion prior is fed into a task-agnostic Motion Tracking Policy trained via Reinforcement Learning (PPO + Adversarial Motion Prior). This policy outputs torque targets for a PD controller inside a physics simulator, ensuring the humanoid agent obeys gravity, maintains balance, and avoids unrealistic artifacts like foot-sliding.
Figure 2. Overview of the PHYLOMAN framework. PHYLOMAN synthesizes behaviors via task planning (LLM →BehaviorScript) and motion planning (Diffusion-guided Control Policy →Humanoid action sequence), achieving goal-oriented, long-term humanoid be- havior generation. The input behavior descriptions are analyzed by a Behavior Planning Network (i.e., LLMs) to produce BehaviorScripts. We then project them into humanoid action space using motion diffusers, thereby guiding the Motion Tracking Policy to control the simu- lated humanoids.

Datasets

The paper utilizes two primary datasets for training and evaluation:

  • GBC-100K: A newly introduced, large-scale benchmark specifically designed for behavior generation. It contains 123.7K sequences (approx. 250 hours) curated from internet videos and academic datasets (Kinetics, UCF101, ActivityNet). Crucially, it features multi-level, hierarchical textual annotations mapping high-level goals to fine-grained actions.
  • HumanML3D: A standard motion-language dataset used to benchmark and evaluate PHYLOMAN against existing short-horizon motion generation baselines.
Figure 1. Overview of the GBC-100K collection process. We collect raw videos, filter for single full-body clips, then apply motion estimation and captioning to extract motions and annotations (a). Data are organized hierarchically (b) with behavior scripts, PoseScripts/- motions, and SMPL sequences.

Key Insights and Ideas

  • From "How" to "Why": Shifting the paradigm from pure motion generation to behavior generation forces models to reason about intentionality and long-term planning, not just immediate kinematics.
  • Parallel-in-Time Generation: Decoupling keyframes and transitions while jointly training them allows the diffusion model to leverage GPU parallelism across the temporal axis, making multi-minute sequence generation computationally tractable.
  • Structured Fine-Tuning: Fine-tuning open-source LLMs (like Qwen-2.5 or Llama-3.1) on the hierarchical annotations of GBC-100K allows them to outperform much larger closed-source models at generating contextually appropriate behavioral trajectories.

Key Results

  • 10× Longer Horizons: PHYLOMAN successfully generates coherent human behaviors over horizons up to 10× longer than existing motion synthesis methods.
  • Massive Success Rate Boost: In long-sequence evaluations (1024 frames), incorporating the TAMP pipeline led to a 133% increase in task success rate (jumping from ~0.3 to over 0.7) compared to baselines without hierarchical planning.
  • Drastic Error Reduction: Compared to methods without TAMP, PHYLOMAN reduced physical errors by 91% (from 1.224 to 0.105). Furthermore, incorporating the physics simulator reduced physical errors by 60.5% over kinematic-only variants while maintaining high semantic diversity.

Limitations and Future Work

Despite its success, PHYLOMAN currently operates with limited environmental awareness. It is trained on datasets with clean text and lacks complex human-scene or human-object interactions. As a result, the model struggles in highly dynamic, contact-rich scenarios (such as swimming or precise tool manipulation).

The authors note that future work will focus on extending PHYLOMAN to open-domain, embodied AI settings that feature complex human-scene interactions and free-form, unscripted instructions.

Summary updated

Abstract

Human motion generative modeling aims to synthesize complex motions from everyday activities. However, current research remains fragmented, focusing either on low-level, short-horizon movements or on high-level, disembodied action planning—thus overlooking the hierarchical and goal-driven nature of human behavior. This paper shifts the focus from mere motion generation to the more comprehensive problem of humanoid behavior modeling. We introduce Generative Behavior Control (GBC), a new task that emphasizes the generation of long-horizon, physically plausible, and semantically coherent behaviors from high-level intentions. To address this task, we propose PHYLOMAN, a hierarchical framework that integrates large language model (LLM)-based task planning with physics-informed motion control under the principles of task and motion planning (TAMP). In parallel, we construct GBC-100K, a large-scale dataset containing hierarchical semantic and motion annotations designed to bridge the gap between abstract goals and executable actions. Extensive experiments demonstrate that PHYLOMAN, trained on GBC-100K, generates more purposeful and natural human behaviors with up to 10× longer temporal horizons than existing methods. This work establishes a foundation for future research on behavior-centric generative modeling, and both code and data will be released to foster further progress.
BibTeX
@inproceedings{zhang2026phyloman,
  author = {Jusheng Zhang and Jinzhou Tang and Sidi Liu and Jian Wang and Keze Wang},
  title = {{PHYLOMAN: Generative Behavior Control via Fusing LLM Planning and Physics-Based Control}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=utSGMkGK1p},
}
Figure