TL;DR
This paper introduces INFERACT, a novel physics-based framework that jointly optimizes human motion and scene layout generation. By leveraging reinforcement learning and explicit physical constraints, INFERACT creates plausible scenes that enable realistic human-object interactions, overcoming limitations of previous kinematics-based methods.
What Problem Does This Paper Address?
Creating realistic 3D animations for movies and video games often requires characters to interact naturally with their environment. However, motion capture data is typically recorded in a blue-screened studio without actual objects. This necessitates animators to manually select and position furniture and objects to match the captured motion, a tedious process prone to errors like object penetration or characters floating. Existing automated methods often struggle with physical plausibility and generalization to diverse motions due to a lack of explicit physical constraints or reliance on specific training data for object affordances. This paper aims to automate the generation of physically plausible scene layouts that seamlessly integrate with given human motions.
Methods
The core of INFERACT is a joint optimization framework powered by reinforcement learning (RL). It simultaneously trains two key components:
- Motion Imitation Controller: This component learns to animate a simulated character within a physics engine to accurately track a given reference human motion. It's trained using a motion tracking reward function that penalizes deviations in root position, orientation, velocity, and joint states.
- Scene Layout Generator: This component predicts suitable objects and their placements to facilitate the character's interaction. It's also trained using RL, aiming to maximize the motion tracking reward.
To enhance the scene layout generation, the authors introduce several novel techniques:
- Physics-based Simulation: By simulating the character in a physics engine, the framework inherently enforces physical constraints, preventing penetration and floating.
- Contact Constraint in Reward: The reward function for the scene layout generator is modified to heavily favor scenes where the character actually makes contact with objects, promoting interaction.
- Pseudo Contact Labels and Pose Priors: Instead of relying on pre-trained contact estimators, the paper proposes an unsupervised method to infer "pseudo contact labels." This is achieved by training a "frame value network" that predicts the tracking performance given a pose and a contact query. By comparing performance with and without contact, pseudo contact labels are derived. These labels are then used to extract "pose priors" (specifically, pelvis poses of contact frames) which provide stronger guidance for object placement, helping to avoid "hacks" where the character might cheat to achieve a good tracking score.
- Dual Optimization: The motion imitation controller and scene layout generator are trained together, allowing them to mutually benefit and converge to a more optimal solution.
Datasets
The paper evaluates its method on the following datasets:
- SAMP: Used for indoor motion capture data.
- PROX: Another dataset for indoor human-object interaction.
- Outdoor vaulting motion capture data: Used to demonstrate generalization to non-indoor scenarios.
The object set used for scene generation is derived from the 3D-Future dataset, comprising 32 furniture items. The authors also tested with other 3D shapes like fountains and rocks to assess generalization to objects without explicit interaction affordances.
Key Insights and Ideas
- Physics as a Constraint, Not Just an Outcome: The most significant insight is the explicit integration of a physics simulator into the scene generation loop. This moves beyond kinematics-based methods that often require post-hoc refinement, ensuring physical plausibility from the ground up.
- Unsupervised Contact Inference: Developing a method to infer contact information (pseudo contact labels) without requiring pre-trained, dataset-specific contact estimators is a crucial step towards greater generalization. This allows the system to work with novel motions and object types.
- Pose Priors as a Solution to RL "Hacks": The use of pose priors derived from pseudo contact labels effectively guides the scene layout generator, preventing the RL agent from finding "cheating" solutions that might achieve a good tracking score but result in unrealistic interactions (e.g., leaning on a chair armrest instead of sitting).
- Joint Optimization Synergy: The simultaneous optimization of the character controller and scene generator creates a positive feedback loop, where a better scene improves motion tracking, which in turn provides better feedback for scene generation.
Key Results
- Quantitative Comparison: The method achieves competitive or superior tracking scores and success rates compared to state-of-the-art methods like SUMMON and MIME on the SAMP dataset. For instance, "Ours" achieved a tracking score of 0.67 and a success rate of 0.84, outperforming SUMMON (0.53 tracking score) and SUMMON in physics (0.61 tracking score, 0.68 success rate).
- Qualitative Results: Visualizations demonstrate that INFERACT generates physically plausible scenes with realistic human-object interactions, avoiding artifacts like penetration and floating seen in previous methods.
- Diversity and Generalization: The method shows the ability to generate diverse scene configurations for a single motion and select appropriate objects for different motions. It also demonstrates generalization to outdoor motions like vaulting, even with objects like tables that may not have obvious interaction affordances.
Limitations and Future Work
- Rigid Body Contact Approximation: The paper acknowledges that approximating all physical interactions using rigid body contact might not fully capture the complex dynamics of real-world human-scene interactions, potentially leading to minor visual artifacts like motion jitters.
- Limited Interaction Scope: The current framework's supported interaction relationships are somewhat restricted.
- Future Directions: The authors suggest exploring more comprehensive optimization objectives to encompass a wider range of interaction categories and potentially incorporating more sophisticated physics models beyond rigid body contact.
Summary updated