TR-LLM: Integrating Trajectory Data for Scene-Aware LLM-Based Human Action Prediction

Kojiro Takeyama, Yimeng Liu, Misha Sra

TL;DR

TR-LLM is a multimodal framework that combines the semantic reasoning of Large Language Models (LLMs) with physical trajectory data to predict human actions in home environments. By merging "what makes sense" (linguistic context) with "where the person is actually going" (physical movement), the system significantly outperforms traditional models, especially when environmental information is incomplete.

What Problem Does This Paper Address?

Predicting what a person will do next is a holy grail for assistive robotics. If a robot knows you’re heading to the microwave to reheat coffee, it can proactively clear a path or prepare the next step of a task.

However, current methods face a "context gap":

  1. Video-based models are limited by the camera's field of view and often miss the "why" behind an action (e.g., is the person going to the fridge because it's breakfast time or because they are cleaning?).
  2. LLMs have vast "common sense" about human routines but are physically "blind." They might suggest a plausible action that is spatially impossible given where the person is currently standing or moving.

TR-LLM bridges this gap by integrating semantic context (time of day, history, persona) with physical constraints (real-time walking trajectories).

Methods

The TR-LLM framework operates in two primary stages:

  1. Target Object Prediction:
    • Semantic Branch (LLM): The model receives a text prompt describing the scene (e.g., "It's 8 AM, the user just finished breakfast"). The LLM ranks potential target objects (fridge, sink, etc.) based on likelihood.
  2. Physical Branch (Trajectory): A dedicated U-Net model analyzes the person's past walking path and a 2D map of the room. It generates a probability heat map of where the person is likely headed.
  3. Integration: The system multiplies these two probability distributions. This effectively filters out objects that are semantically likely but physically unreachable, and vice versa.
  4. Action Prediction: Once the most likely target object is identified, the LLM generates the specific action (e.g., "reheating the coffee" vs. "cleaning the microwave").

Datasets

The researchers utilized and created several key resources:

  • LocoVR: A dataset of over 7,000 human trajectories in 131 indoor environments, used to train the U-Net trajectory predictor.
  • HM3DSem (Habitat-Matterport 3D Semantics): Used to extract realistic 3D home layouts and object locations.
  • New Evaluation Benchmark: The authors built a custom evaluation dataset consisting of 67 data pairs that combine scene maps, trajectories, and complex semantic scenarios (like "morning routines" or "weekend relaxation").

Key Insights and Ideas

  • Trajectory as a Filter: The "aha moment" is that physical movement acts as a powerful noise filter for LLMs. An LLM might hallucinate that a user is going to the stove in the morning, but if the trajectory shows them walking toward the door, the integrated model corrects the prediction in real-time.
  • Robustness to "Information Blackouts": One of the most striking findings is that TR-LLM remains accurate even when semantic information (like conversation history) is missing. The trajectory data "picks up the slack," making the system more reliable in messy, real-world deployments.
  • The "Progress Effect": Prediction accuracy increases naturally as the person moves. The closer a person gets to an object, the more the physical trajectory narrows down the possibilities, compensating for any initial semantic ambiguity.

Key Results

The authors tested TR-LLM against standalone LLMs (GPT-4o-mini, Llama 3.2) and Vision-Language Models (VLMs):

  • Superior Accuracy: TR-LLM (using GPT-4o-mini) achieved a target object prediction accuracy of 58.6%, significantly higher than the standalone LLM (44.7%) or the standalone trajectory model (39.7%).
  • VLM Limitations: Interestingly, pre-trained VLMs struggled with the task. While good at identifying objects in images, they lacked the specialized spatial reasoning required to predict future destinations based on movement patterns.
  • Action Prediction: In predicting the final action (measured by cosine similarity of text descriptions), TR-LLM consistently outperformed all baselines, proving that getting the object right is the most critical step in getting the action right.

Limitations and Future Work

  • Locomotion Only: The current model only looks at walking trajectories. The authors acknowledge that "full-body motion" (hand gestures, head orientation, and gaze) would provide even richer cues for action prediction.
  • Scale: The study focused on 10m x 10m home environments. Future work is needed to see how these models scale to larger, more crowded public spaces like malls or airports.
  • Real-time Latency: While the paper discusses training times, the practical constraints of running these multi-step LLM chains on a mobile robot in real-time remain an area for further optimization.

Summary updated

Abstract

Accurate prediction of human behavior is crucial for AI systems to effectively support real-world applications, such as autonomous robots anticipating and assisting with human tasks. Real-world scenarios frequently present challenges such as occlusions and incomplete scene observations, which can compromise predictive accuracy. Thus, traditional video-based methods often struggle due to limited temporal and spatial perspectives. Large Language Models (LLMs) offer a promising alternative. Having been trained on a large text corpus describing human behaviors, LLMs likely encode plausible sequences of human actions in a home environment. However, LLMs, trained primarily on text data, lack inherent spatial awareness and real-time environmental perception. They struggle with understanding physical constraints and spatial geometry. Therefore, to be effective in a real-world spatial scenario, we propose a multimodal prediction framework that enhances LLM-based action prediction by integrating physical constraints derived from human trajectories. Our experiments demonstrate that combining LLM predictions with trajectory data significantly improves overall prediction performance. This enhancement is particularly notable in situations where the LLM receives limited scene information, highlighting the complementary nature of linguistic knowledge and physical constraints in understanding and anticipating human behavior.
BibTeX
@inproceedings{takeyama2025tr,
  author = {Kojiro Takeyama and Yimeng Liu and Misha Sra},
  title = {{TR-LLM: Integrating Trajectory Data for Scene-Aware LLM-Based Human Action Prediction}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246714},
}

Loading related content...

Figure