Abstract
Generating realistic human motion within 3D environments requires a model to understand both the physics of movement and the semantic relationship between text commands and scene geometry. Current state-of-the-art models often struggle with "grounding"—accurately linking language to specific objects in space—leading to "spatial drift" or "center bias." We introduce GHOST (Grounded Human Motion Generation with Open Vocabulary Scene-and-Text Contexts), a framework that leverages open-vocabulary vision-language models and specialized spatial losses. By aligning 3D scene features with CLIP text embeddings through knowledge distillation, GHOST reduces the distance between generated humans and their goal objects by up to 30% compared to existing baselines.
Introduction
The core challenge in text-to-motion generation is grounding. Previous models, such as HUMANISE, typically utilize "closed vocabulary" scene encoders trained on fixed object categories. When faced with complex or free-form text prompts, these models often fail to connect the text to specific 3D coordinates. This results in characters "sitting" in empty space or defaulting to the center of a scene. GHOST addresses these issues by moving to an open-vocabulary approach, ensuring the model can interpret a wider range of descriptions and accurately locate targets within a 3D environment.
Methods: Open Vocabulary and Grounding Losses
The GHOST framework employs a two-step approach to improve spatial accuracy:
- Open Vocabulary Distillation: Instead of training a 3D scene encoder from scratch, GHOST distills knowledge from 2D open-vocabulary models (e.g., LSeg or OpenSeg). By projecting 3D points into 2D camera views, the 3D encoder (a Point Transformer U-Net) is trained to produce features matching the 2D model's CLIP-aligned embeddings. This creates a shared "language" between the 3D scene and the text prompt.
- Fine-tuning with Grounding Losses: The model uses a Conditional Variational Autoencoder (cVAE) to generate motion. To sharpen grounding, two novel regularization losses are introduced: $\mathcal{L}_{bbox}$ (regressing 3D bounding box corners) and $\mathcal{L}_{class}$ (classifying the goal object category). These losses force the model to be aware of the target object's size and location.
Experimental Setup and Datasets
The primary evaluation was conducted using the HUMANISE dataset, which contains nearly 20,000 motion sequences across 643 3D scenes from ScanNet. The dataset includes template-based text annotations describing spatial relations. Additionally, the authors tested zero-shot capabilities using free-form text to evaluate the generalization of the open-vocabulary approach.
Results and Discussion
GHOST demonstrates significant improvements across both quantitative and qualitative metrics:
- Quantitative Accuracy: GHOST achieved a 30% reduction in "Goal Object Distance." For "walk" actions, the error dropped from 1.37m (baseline) to 0.95m.
- Qualitative Alignment: Visualizations confirm that GHOST places characters much closer to targets and handles complex spatial relations (e.g., "farthest from") more effectively than the HUMANISE baseline.
- Human Perception: In a user study, participants preferred GHOST-generated motions 63.27% of the time, noting better alignment with textual descriptions.
Conclusion and Future Work
GHOST successfully bridges the gap between 3D scene geometry and open-vocabulary text prompts. By utilizing CLIP-aligned features and explicit grounding losses, it sets a new standard for spatial accuracy in human motion generation. Future work will focus on reducing scene penetration (clipping through furniture) and improving fine-grained orientation. The authors also suggest exploring Diffusion Models to increase motion diversity and incorporating contact optimization for more realistic interactions.
Summary updated