GHOST: Grounded Human Motion Generation with Open Vocabulary Scene-and-Text Contexts

Zoltan A. Milacski, Koichiro Niinuma, Ryosuke Kawamura, Fernando De la Torre, László A. Jeni

TL;DR

GHOST (Grounded Human Motion Generation) significantly improves how AI-generated characters interact with 3D environments by aligning text and scene features in a shared "open vocabulary" space. By distilling knowledge from 2D vision-language models, GHOST achieves up to a 30% reduction in grounding errors, ensuring characters actually "walk to the chair" rather than just wandering into the middle of the room.

What Problem Does This Paper Address?

Generating realistic human motion in a 3D scene—like "sitting on the sofa near the window"—requires the AI to master three distinct modalities: natural language, 3D geometry, and human kinematics.

Current state-of-the-art models, such as the HUMANISE framework, often struggle with grounding. They typically use "closed vocabulary" scene encoders that only recognize a fixed set of object categories (e.g., "chair," "table"). Because the text encoder (usually CLIP-based) is "open vocabulary" and can describe almost anything, there is a fundamental misalignment between what the AI reads and what it "sees" in 3D. This often results in a "center bias," where the model, unsure of where the specific target object is, simply places the character in the middle of the scene.

Fig. 1: Comparison of our proposed GHOST cVAE method with the prior state-of-the- art HUMANISE cVAE [71] in text-and-scene-conditional human motion generation. Best viewed in color. (a) The HUMANISE cVAE exhibits a bias towards generating motions centered within the scene. (b) In contrast, our GHOST cVAE demonstrates superior semantic understanding and achieves higher action performance. (c) The three implementations of our GHOST framework exhibit approximately 1.5× to 3.9× larger parameter counts (indicated by dot radii) than the HUMANISE cVAE. All of our three variants outperform the baseline in two text-scene grounding metrics.

Methods

The authors introduce a two-step approach to bridge the gap between text and 3D scenes:

  1. Open Vocabulary Pretraining (Distillation): Instead of training a 3D scene encoder from scratch, the authors use knowledge distillation. They take a Point Transformer U-Net and train it to mimic the features of 2D open-vocabulary segmentation models (like LSeg or OpenSeg). Since these 2D models are already aligned with CLIP’s text space, the 3D encoder inherits a "shared language" between pixels, points, and words.
  2. Fine-tuning with Grounding Regularization: During the motion generation phase (using a Conditional Variational Autoencoder or cVAE), the model is fine-tuned with two novel auxiliary losses:
  3. Bounding Box Regression ($L_{bbox}$): Forces the model to predict the corners of the target object, making it more aware of the object's physical size and extent.
  4. Category Classification ($L_{class}$): Reinforces semantic understanding by classifying the goal object.
Fig. 2: Overview of our idea. Best viewed in color. We compare our GHOST cVAE with the HUMANISE cVAE [71] model. The major differences are in the text and 3D scene point cloud representations, grounding and regularization. (a) The HUMANISE cVAE architecture utilizes a closed vocabulary scene encoder producing a finite set of labels, resulting in a misalignment with the open vocabulary text feature space. This requires the fusion module to learn grounding from scratch. Grounding is regularized by regressing the center point of the goal object. (b) In contrast, our GHOST cVAE archi- tecture employs a shared open vocabulary vision-language space for both modalities, establishing initial grounding between them. We regularize grounding by classifying and regressing the bounding box corners of the goal object, increasing awareness for category and size.

Datasets

The researchers evaluated GHOST on the HUMANISE dataset, currently the largest and most diverse benchmark for text-and-scene-conditional motion. It consists of:

  • 19,648 motion sequences from the AMASS collection.
  • 643 3D scenes from ScanNet.
  • Template-based text annotations (e.g., "walk to the [goal] that is [spatial relation] the [anchor]").

Key Insights and Ideas

  • Leveraging 2D for 3D: The most powerful insight is that we don't need massive 3D-text datasets to learn grounding. By distilling knowledge from 2D vision-language models (which are trained on billions of internet images), we can "gift" 3D models a sophisticated understanding of the world.
  • Size Matters for Grounding: Moving beyond simple "center point" regression to "bounding box" regression was crucial. Knowing the size of a bed versus a chair helps the model generate more plausible approach and interaction motions.
  • Open Vocabulary vs. Closed: The shift to an open-vocabulary scene encoder allows the model to handle much more descriptive and varied text prompts without needing to retrain the entire scene understanding component for every new object type.

Key Results

  • Quantitative: GHOST achieved a 30% reduction in the "Goal Object Distance" metric (the physical distance between the generated human and the target object) compared to the HUMANISE baseline.
  • Qualitative: As seen in the attention maps, GHOST successfully identifies specific instances of objects (e.g., a specific chair among many) based on spatial descriptions, whereas the baseline often defaults to the scene center.
  • Perceptual Study: In a user study with 27 participants, GHOST’s generated motions were preferred over the baseline 63.27% of the time, with users noting better alignment with the text descriptions.
  • Efficiency: Despite having 1.5x to 3.9x more parameters, motion sampling remains fast, taking only ~0.19s on an A100 GPU.
Qualitative results showing GHOST correctly identifying target objects in complex scenes.

Limitations and Future Work

While GHOST is a major step forward, the authors acknowledge several remaining hurdles:

  • Fine-grained Errors: The model still occasionally struggles with precise body orientation and "scene penetration" (e.g., a leg clipping through a coffee table).
  • Teacher Dependency: The performance is inherently capped by the quality of the 2D teacher models used for distillation.
  • Future Directions: The authors suggest exploring Diffusion Models (which have shown great promise in image and motion generation) as a replacement for the cVAE architecture to further improve motion diversity and quality. They also aim to incorporate better contact optimization to prevent physical clipping.

Summary updated

Abstract

The connection between our 3D surroundings and the descriptive language that characterizes them would be well-suited for localizing and generating human motion in context but for one problem. The complexity introduced by multiple modalities makes capturing this connection challenging with a fixed set of descriptors. Specifically, closed vocabulary scene encoders, which require learning text-scene associations from scratch, have been favored in the literature, often resulting in inaccurate motion grounding. In this paper, we propose a method that integrates an open vocabulary scene encoder into the architecture, establishing a robust connection between text and scene. Our two-step approach starts with pretraining the scene encoder through knowledge distillation from an existing open vocabulary semantic image segmentation model, ensuring a shared text-scene feature space. Subsequently, the scene encoder is fine-tuned for conditional motion generation, incorporating two novel regularization losses that regress the category and size of the goal object. Our methodology achieves up to a 30% reduction in the goal object distance metric compared to the prior state-of-the-art baseline model on the HUMANISE dataset. This improvement is demonstrated through evaluations conducted using three implementations of our framework and a perceptual study. Additionally, our method is designed to seamlessly accommodate future 2D segmentation methods that provide per-pixel text-aligned features for distillation.
BibTeX
@inproceedings{milacski2025ghost,
  author = {Zoltan A. Milacski and Koichiro Niinuma and Ryosuke Kawamura and Fernando De la Torre and László A. Jeni},
  title = {{GHOST: Grounded Human Motion Generation with Open Vocabulary Scene-and-Text Contexts}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00404},
}
Figure