TL;DR
This paper introduces GHOST, a novel framework for generating human motion in 3D scenes based on both text descriptions and the scene's visual context. By leveraging open-vocabulary scene understanding and a two-step pretraining/fine-tuning approach, GHOST significantly improves the accuracy of grounding human actions to specific objects within a scene, outperforming previous methods.
What Problem Does This Paper Address?
Generating realistic and contextually appropriate human motion in 3D environments is crucial for applications like virtual reality, gaming, and robotics. A key challenge is accurately grounding these motions to specific objects and spatial relationships described in text, within complex 3D scenes. Existing methods often struggle with this due to limitations in how they process scene information, frequently relying on "closed vocabulary" scene encoders that can't handle the diversity of natural language descriptions. This leads to motions that are not precisely located relative to the intended goal object, often being biased towards the center of the scene.
Methods
GHOST tackles this by introducing an "open vocabulary" approach to scene understanding. The core idea is to create a shared feature space between text and 3D scene representations. This is achieved in two main stages:
- Pretraining: The 3D scene encoder is pretrained using knowledge distillation from an existing open-vocabulary semantic image segmentation model. This aligns the scene encoder's features with those of a text encoder (specifically, CLIP), ensuring a common understanding of visual and linguistic concepts.
- Fine-tuning: The pretrained scene encoder is then fine-tuned for the motion generation task. This stage incorporates two novel regularization losses: one that regresses the bounding box of the goal object and another that classifies its category. These losses help the model better understand the goal object's location, size, and identity.
The overall architecture uses a Conditional Variational Autoencoder (cVAE) for motion generation, similar to prior work, but with the enhanced scene encoder and grounding mechanisms.
Datasets
The primary dataset used for training and evaluation is HUMANISE. This dataset consists of:
- 19,648 AMASS motion sequences.
- Synthetically aligned with 643 ScanNet 3D scenes.
- Approximately 1.2 million motion frames in total.
- Motions correspond to four BABEL actions: walk, sit, stand up, and lie.
- Text annotations follow a template-based format describing actions, goal objects, and spatial relations.
The authors also mention using open-vocabulary image segmentation models (LSeg, OpenSeg, OVSeg) for knowledge distillation during pretraining.
Key Insights and Ideas
- Open Vocabulary is Key: The core insight is that using open-vocabulary scene encoders, which can understand a wider range of objects and descriptions, is crucial for bridging the gap between language and 3D scene context in motion generation.
- Shared Feature Space: Aligning text and 3D scene features in a common space (like CLIP's) before motion generation is more effective than trying to learn this alignment from scratch during the motion generation phase.
- Beyond Center Bias: The HUMANISE baseline often places motions near the scene's center. GHOST's grounding mechanisms, particularly the bounding box regression, help overcome this bias and focus on the specific goal object.
- Plug-and-Play Teacher Models: The pretraining framework is designed to be flexible, allowing integration with various open-vocabulary segmentation models as "teachers."
Key Results
- Improved Grounding: GHOST significantly reduces the distance between the generated motion and the goal object. The best performing variant (GHOST OpenSeg) achieved up to a 41.8 cm reduction on the "walk" subset and 27.6 cm on the entire dataset compared to the HUMANISE CVAE baseline.
- Perceptual Preference: In a user study, participants preferred motions generated by GHOST OpenSeg over the baseline 63.27% of the time, unanimously selecting GHOST's outputs.
- Ablation Study: The study confirmed the importance of aligning text and scene encoders. Regularization losses also showed significant impact, with bounding box loss being particularly crucial.
- Computational Efficiency: While the scene encoder is larger, motion sampling time is only slightly increased (1.3x longer wall-clock time).
Limitations and Future Work
- Remaining Errors: The paper acknowledges that GHOST still exhibits some errors in goal identification, orientation, and scene penetration.
- Future Directions:
- Investigating better Vision-Language Models (VLMs) and teacher models.
- Exploring more advanced regularization techniques.
- Substituting the cVAE with a diffusion model for motion generation.
- Extending grounding to non-goal/non-anchor objects.
- Implementing post-processing for contact optimization.
- Addressing generalization to more natural language and a wider range of actions.
Summary updated