TL;DR
GHOST (Grounded Human Motion Generation) significantly improves how AI-generated characters interact with 3D environments by aligning text and scene features in a shared "open vocabulary" space. By distilling knowledge from 2D vision-language models, GHOST achieves up to a 30% reduction in grounding errors, ensuring characters actually "walk to the chair" rather than just wandering into the middle of the room.
What Problem Does This Paper Address?
Generating realistic human motion in a 3D scene—like "sitting on the sofa near the window"—requires the AI to master three distinct modalities: natural language, 3D geometry, and human kinematics.
Current state-of-the-art models, such as the HUMANISE framework, often struggle with grounding. They typically use "closed vocabulary" scene encoders that only recognize a fixed set of object categories (e.g., "chair," "table"). Because the text encoder (usually CLIP-based) is "open vocabulary" and can describe almost anything, there is a fundamental misalignment between what the AI reads and what it "sees" in 3D. This often results in a "center bias," where the model, unsure of where the specific target object is, simply places the character in the middle of the scene.
Methods
The authors introduce a two-step approach to bridge the gap between text and 3D scenes:
- Open Vocabulary Pretraining (Distillation): Instead of training a 3D scene encoder from scratch, the authors use knowledge distillation. They take a Point Transformer U-Net and train it to mimic the features of 2D open-vocabulary segmentation models (like LSeg or OpenSeg). Since these 2D models are already aligned with CLIP’s text space, the 3D encoder inherits a "shared language" between pixels, points, and words.
- Fine-tuning with Grounding Regularization: During the motion generation phase (using a Conditional Variational Autoencoder or cVAE), the model is fine-tuned with two novel auxiliary losses:
- Bounding Box Regression ($L_{bbox}$): Forces the model to predict the corners of the target object, making it more aware of the object's physical size and extent.
- Category Classification ($L_{class}$): Reinforces semantic understanding by classifying the goal object.
Datasets
The researchers evaluated GHOST on the HUMANISE dataset, currently the largest and most diverse benchmark for text-and-scene-conditional motion. It consists of:
- 19,648 motion sequences from the AMASS collection.
- 643 3D scenes from ScanNet.
- Template-based text annotations (e.g., "walk to the [goal] that is [spatial relation] the [anchor]").
Key Insights and Ideas
- Leveraging 2D for 3D: The most powerful insight is that we don't need massive 3D-text datasets to learn grounding. By distilling knowledge from 2D vision-language models (which are trained on billions of internet images), we can "gift" 3D models a sophisticated understanding of the world.
- Size Matters for Grounding: Moving beyond simple "center point" regression to "bounding box" regression was crucial. Knowing the size of a bed versus a chair helps the model generate more plausible approach and interaction motions.
- Open Vocabulary vs. Closed: The shift to an open-vocabulary scene encoder allows the model to handle much more descriptive and varied text prompts without needing to retrain the entire scene understanding component for every new object type.
Key Results
- Quantitative: GHOST achieved a 30% reduction in the "Goal Object Distance" metric (the physical distance between the generated human and the target object) compared to the HUMANISE baseline.
- Qualitative: As seen in the attention maps, GHOST successfully identifies specific instances of objects (e.g., a specific chair among many) based on spatial descriptions, whereas the baseline often defaults to the scene center.
- Perceptual Study: In a user study with 27 participants, GHOST’s generated motions were preferred over the baseline 63.27% of the time, with users noting better alignment with the text descriptions.
- Efficiency: Despite having 1.5x to 3.9x more parameters, motion sampling remains fast, taking only ~0.19s on an A100 GPU.
Limitations and Future Work
While GHOST is a major step forward, the authors acknowledge several remaining hurdles:
- Fine-grained Errors: The model still occasionally struggles with precise body orientation and "scene penetration" (e.g., a leg clipping through a coffee table).
- Teacher Dependency: The performance is inherently capped by the quality of the 2D teacher models used for distillation.
- Future Directions: The authors suggest exploring Diffusion Models (which have shown great promise in image and motion generation) as a replacement for the cVAE architecture to further improve motion diversity and quality. They also aim to incorporate better contact optimization to prevent physical clipping.
Summary updated