TL;DR
This paper introduces PDP, a novel diffusion policy for physics-based character animation that combines reinforcement learning (RL) and behavior cloning (BC) to achieve robust and diverse motion generation. PDP excels at tasks like perturbation recovery, motion tracking, and text-to-motion synthesis by leveraging a "noisy-state, clean-action" data sampling strategy to improve generalization and handle unseen scenarios.
What Problem Does This Paper Address?
Generating realistic and physically interactive human motion for character animation is a long-standing challenge. While diffusion models have shown promise in capturing diverse motion patterns, naively applying them to high-frequency, under-actuated control tasks like bipedal locomotion leads to unstable motions due to compounding errors. Existing methods often struggle to balance motion diversity with robustness against environmental changes or simulation inaccuracies. This paper aims to bridge this gap by developing a diffusion policy that is not only capable of generating diverse motions but also robust to perturbations and domain shifts, making it suitable for complex physics-based applications.
Methods
PDP employs a three-stage approach:
- Expert Policy Training: The process begins by training a set of specialized expert reinforcement learning (RL) policies, each designed to handle a subset of tasks. These experts are trained to perform specific motion tracking tasks within a physics simulator.
- Stochastic Data Collection: To address the robustness issue, PDP utilizes a novel data sampling strategy called "noisy-state, clean-action." Instead of collecting perfect state-action pairs, this method introduces noise to the observed states while retaining the expert's optimal action. This "noise band" around clean trajectories effectively expands the training data distribution, forcing the policy to learn corrective actions for sub-optimal states. This is inspired by the DASS strategy but further enhanced by generating short recovery episodes from random initial poses.
- Behavior Cloning with Diffusion Policy: Finally, a single diffusion policy is trained using behavior cloning (BC) on the collected stochastic dataset. The policy is implemented as a transformer encoder-decoder architecture, similar to Diffusion Policy, but with modifications for different applications (e.g., incorporating CLIP for text-to-motion). The diffusion model learns to denoise action sequences conditioned on observations and task information.
Datasets
The paper evaluates PDP on several datasets:
- Bump'em dataset: A subset of the Addbiomechanics dataset, used for training and evaluating perturbation recovery during locomotion. This dataset features human participants being pushed while walking, providing diverse recovery strategies.
- AMASS dataset: Used for universal motion tracking. The paper utilizes standard train/test splits from existing work (PHC).
- KIT subset of AMASS and HumanML3D: Used for physics-based text-to-motion synthesis. HumanML3D provides text annotations for motion capture data from the KIT dataset.
No new benchmarks were introduced; existing datasets were leveraged to demonstrate PDP's versatility.
Key Insights and Ideas
- "Noisy-state, clean-action" is crucial for robustness: The core insight is that RL policies can provide corrective actions from sub-optimal states. By pairing noisy observations with clean actions during training, the diffusion policy learns to recover from errors and perturbations, significantly improving generalization.
- Diffusion models can capture multi-modality in physics-based control: Unlike deterministic methods (like MLPs) that might average out responses, diffusion models can effectively learn and represent diverse recovery strategies for the same perturbation, leading to more human-like and robust behavior.
- Sampling strategy matters more than just data quantity: The paper demonstrates that the way data is collected (noisy vs. clean states/actions) has a more significant impact on robustness than simply having more data, especially for challenging tasks like perturbation recovery.
- PDP offers a unified framework: The method is shown to be versatile, applicable to diverse tasks (perturbation recovery, motion tracking, text-to-motion) with minimal architectural changes, relying on different conditioning mechanisms.
Key Results
- Perturbation Recovery: PDP achieved a 100% success rate on in-distribution perturbations and a 96.3% success rate on out-of-distribution perturbations, significantly outperforming C-VAE (91.3%) and MLP baselines. It also achieved a much lower foot placement correctness (FPC) score (2.79) compared to C-VAE (4.97), indicating more accurate foot placement.
- Motion Tracking: PDP achieved a 97.1% success rate on the AMASS test dataset, matching or exceeding state-of-the-art methods like PHC (96.4%) across various metrics (Eg-mpjpe, Empjpe, Evel, Eacc).
- Text-to-Motion: PDP demonstrated the ability to follow diverse text commands, achieving a 57.1% success rate, significantly outperforming an MLP baseline (11.9%).
- Sampling Strategy: The "noisy-state, clean-action" strategy yielded the best results, achieving 100% success in perturbation recovery and 93.5% in tracking, outperforming "clean-clean" and "noisy-noisy" strategies.
Limitations and Future Work
- Inference Speed: The primary limitation is the inference speed of diffusion models, which is K times slower than MLPs (where K is the number of denoising steps). Future work could explore methods to reduce inference time, such as those mentioned by [Huang et al. 2024] and [Yin et al. 2023].
- Long-Horizon Prediction Trade-off: Predicting over longer horizons for diversity can dilute the importance of immediate actions, potentially reducing robustness. Future work could investigate adaptive weighting schemes to balance long-horizon prediction with immediate action precision.
- Text-to-Motion Nuances: The paper notes that text-to-motion performance is not as high as kinematic generation, partly due to the challenge of balancing motion execution with maintaining equilibrium. Future work could focus on improving this balance.
- Leveraging Pre-trained Models: The authors suggest future work could involve speeding up inference by leveraging large pre-trained motion trackers for downstream RL tasks.
Summary updated