Abstract
Modeling 3D Hand-Object Interaction (HOI) is a complex task essential for robotics, AR/VR, and human-activity understanding. We present HOIGPT, a unified framework that leverages Large Language Models (LLMs) to both perceive and generate 3D HOI sequences. By treating 3D motion as a discrete "language" through a novel decomposed tokenizer, HOIGPT overcomes the limitations of existing diffusion-based models, such as short temporal horizons and physical inconsistencies. Our model achieves state-of-the-art results in HOI generation and captioning, demonstrating the power of applying linguistic reasoning to 3D geometric tasks.
Introduction
Capturing how humans manipulate objects is a cornerstone for modern interactive technologies. However, modeling 3D Hand-Object Interaction (HOI) is notoriously difficult due to the coordinated dance of two hands and an object. Existing approaches, largely based on diffusion models, face three major hurdles: limited flexibility (one-way generation), short temporal horizons that struggle with multi-step tasks, and physical unreality where hands pass through solid objects.
HOIGPT addresses these challenges by bridging the gap between 3D geometry and linguistics. The core insight is that 3D interaction sequences possess a structural hierarchy similar to language. By discretizing these into tokens, the framework can tap into the massive world knowledge and reasoning capabilities present in pre-trained LLMs.
Methodology
The HOIGPT architecture converts 3D motion into discrete tokens that an LLM can process, consisting of two primary components: a decomposed tokenizer and a motion-aware language model.
HOI-Decomposed VQ-VAE
A naive tokenizer would struggle with the combinatorial explosion of two hands and an object. Instead, HOIGPT uses a decomposed approach with separate codebooks for hand motions and object motions. By leveraging symmetry, a single codebook represents both left and right hands, which significantly improves efficiency and generalization. This factorization allows the model to learn "how a hand picks up" and "how an object moves" independently and then combine them creatively.
Motion-Aware Language Model
The researchers utilized Flan-T5-Base as the core reasoning engine, trained in two stages. First, a pre-training phase teaches the model the "grammar" of motion through masked token prediction. Second, instruction tuning fine-tunes the model on specific tasks like text-to-HOI or HOI captioning using natural language prompts. To ensure physical plausibility, a Geometric-Aware Loss penalizes interpenetration and rewards realistic contact and approach trajectories.
Results and Discussion
HOIGPT was evaluated on the ARCTIC and GRAB datasets, achieving state-of-the-art (SOTA) results across multiple benchmarks:
- HOI Generation: The model achieved a Frechet Inception Distance (FID) of 3.29, a massive improvement over the previous SOTA of 5.85. Lower FID indicates generated motion is much closer to real human data.
- Physical Realism: The model significantly reduced interpenetration volume compared to diffusion-based baselines, proving the effectiveness of the geometric loss.
- Long-Sequence Capability: Qualitatively, the model excels at multi-step tasks, such as picking up an item, placing it back, and passing it with the right hand, where previous models often became incoherent during such long sequences.
Conclusion
HOIGPT represents a significant step forward in unified 3D interaction modeling. By treating motion as a foreign language, it enables complex, physically plausible, and long-horizon HOI generation and perception. While challenges remain regarding data scarcity and inference speed for extremely long sequences, the framework provides a robust foundation for future research in multi-object and human-human-object interactions.
Summary updated