Abstract
Accurate prediction of human behavior is crucial for AI systems to effectively support real-world applications, such as autonomous robots anticipating and assisting with human tasks. Real-world scenarios frequently present challenges such as occlusions and incomplete scene observations, which can compromise predictive accuracy. Thus, traditional video-based methods often struggle due to limited temporal and spatial perspectives. Large Language Models (LLMs) offer a promising alternative. Having been trained on a large text corpus describing human behaviors, LLMs likely encode plausible sequences of human actions in a home environment. However, LLMs, trained primarily on text data, lack inherent spatial awareness and real-time environmental perception. They struggle with understanding physical constraints and spatial geometry. Therefore, to be effective in a real-world spatial scenario, we propose a multimodal prediction framework that enhances LLM-based action prediction by integrating physical constraints derived from human trajectories. Our experiments demonstrate that combining LLM predictions with trajectory data significantly improves overall prediction performance. This enhancement is particularly notable in situations where the LLM receives limited scene information, highlighting the complementary nature of linguistic knowledge and physical constraints in understanding and anticipating human behavior.
Introduction
Predicting human behavior is a cornerstone for assistive robotics. If a robot can anticipate that a user is heading to the microwave to reheat coffee, it can proactively assist by clearing a path or preparing the next step of a task. However, current methods face a "context gap." Video-based models are often limited by field-of-view and lack semantic depth, while Large Language Models (LLMs), despite their vast common-sense knowledge, lack physical grounding and spatial awareness. They may suggest actions that are semantically plausible but physically impossible given a person's current location. TR-LLM addresses this by integrating semantic reasoning with physical trajectory data.
Methodology
The TR-LLM framework operates in two primary stages to bridge the gap between semantic context and physical constraints:
- Target Object Prediction:
- Semantic Branch (LLM): The model processes scene context (e.g., time of day, action history, persona) to rank potential target objects based on likelihood.
- Physical Branch (Trajectory): A dedicated U-Net model analyzes the person's past walking path and a 2D map of the room to generate a probability heat map of where the person is likely headed.
- Integration: The system multiplies these two probability distributions. This effectively filters out objects that are semantically likely but physically unreachable, and vice versa.
- Action Prediction: Once the most likely target object is identified, the LLM generates the specific action (e.g., "reheating the coffee" vs. "cleaning the microwave").
Datasets and Evaluation
The researchers utilized several key resources to train and validate the framework:
- LocoVR: A dataset of over 7,000 human trajectories in 131 indoor environments, used to train the U-Net trajectory predictor.
- HM3DSem (Habitat-Matterport 3D Semantics): Used to extract realistic 3D home layouts and object locations.
- New Evaluation Benchmark: A custom dataset of 67 data pairs combining scene maps, trajectories, and complex semantic scenarios (e.g., morning routines).
Results and Discussion
TR-LLM significantly outperforms standalone models. Using GPT-4o-mini, it achieved a target object prediction accuracy of 58.6%, compared to 44.7% for the standalone LLM and 39.7% for the trajectory-only model. Interestingly, pre-trained Vision-Language Models (VLMs) struggled, as they lacked the specialized spatial reasoning required to predict future destinations from movement patterns.
Key insights from the study include:
- Trajectory as a Filter: Physical movement acts as a powerful noise filter, correcting LLM "hallucinations" regarding likely destinations in real-time.
- Robustness to Information Gaps: The model remains accurate even when semantic information (like conversation history) is missing, as trajectory data compensates for the lack of context.
- The Progress Effect: Prediction accuracy naturally increases as the person moves closer to an object, narrowing down the physical possibilities.
Conclusion
TR-LLM demonstrates that combining linguistic common sense with physical constraints significantly improves human action prediction. While currently focused on walking trajectories in home environments, future work will explore integrating full-body motion (gaze, gestures) and scaling the framework to larger, more complex public spaces.
Summary updated