IA-LSTM: Interaction-Aware LSTM for Pedestrian Trajectory Prediction

Jing Yang, Yuehai Chen, Shaoyi Du, Badong Chen, José C. Príncipe

TL;DR

This paper introduces IA-LSTM, a novel approach for predicting pedestrian trajectories in crowded environments. It addresses the challenge of accurately modeling human-human interactions by using correntropy to dynamically weigh interaction importance and define personal space, leading to improved prediction accuracy compared to existing methods.

What Problem Does This Paper Address?

Predicting where pedestrians will move next is crucial for applications like autonomous driving and robotics, enabling safer navigation and decision-making. However, accurately forecasting these movements is difficult due to the complex and dynamic nature of human interactions. Pedestrians don't just move randomly; they influence each other's paths through social behaviors like yielding, walking in groups, and maintaining personal space. Previous methods often treated all interactions equally or relied on handcrafted rules, failing to capture the nuanced importance of different interactions based on proximity and context. This paper aims to bridge this gap by developing a model that can intelligently measure and leverage these varying interaction dynamics.

Methods

The core innovation of this paper is the introduction of an "Interaction Module" within a Long Short-Term Memory (LSTM) network architecture, dubbed IA-LSTM.

  • Correntropy for Interaction Measurement: The key to their approach is the use of correntropy, a measure from information theory, to quantify the importance of interactions between pedestrians. Correntropy is calculated using a Gaussian kernel, which naturally models the concept of "personal space." Interactions within this space are considered more significant, while those outside are down-weighted, effectively treating distant pedestrians as outliers. This allows the model to learn the relative importance of interactions without predefined rules.
  • Interaction Tensor: The calculated correntropy values are used to construct an "Interaction Tensor." This tensor captures the feature representations of dynamic human-human interactions, effectively summarizing how pedestrians are influencing each other at a given time.
  • Interaction-Aware LSTM: The IA-LSTM architecture integrates this Interaction Module. Each pedestrian is processed by an LSTM, but the input to the LSTM is augmented with the information from the Interaction Module. This allows the model to condition each pedestrian's trajectory prediction not only on their own past movements but also on the dynamically weighted interactions with others in the scene.
  • Data-Driven Approach: The model is entirely data-driven, learning interaction patterns directly from observed pedestrian behavior rather than relying on manually defined social force models or interaction rules.

Datasets

The proposed IA-LSTM model was evaluated on two widely used public datasets for pedestrian trajectory prediction:

  • ETH [40]: This dataset includes two scenes (ETH and Hotel).
  • UCY [104]: This dataset includes three scenes (ZARA01, ZARA02, and UCY).

These datasets are noted for containing complex scenarios with group movements, varying velocities, and intentional collision avoidance, making them suitable for testing interaction-aware models.

Key Insights and Ideas

  • Personal Space Matters: The paper highlights that human-human interactions are not uniform; they are heavily influenced by personal space. This concept is crucial for accurate trajectory prediction, and the correntropy-based approach effectively models this.
  • Dynamic Interaction Weighting: Instead of treating all interactions equally, the model learns to assign varying importance to them. This is a significant improvement over methods that might overemphasize distant or irrelevant interactions.
  • Correntropy as a Robust Measure: Correntropy's robustness to outliers and its ability to generalize high-dimensional features make it a suitable tool for capturing complex social dynamics in pedestrian behavior.
  • End-to-End Learning: The IA-LSTM model learns to extract interaction features and predict trajectories in an end-to-end fashion, simplifying the modeling process and potentially leading to more integrated solutions.

Key Results

  • Improved Accuracy: The IA-LSTM model consistently outperformed baseline methods like "Social LSTM" and "Social Attention" on both Average Displacement Error (ADE) and Final Displacement Error (FDE) metrics across various datasets. For instance, it achieved an 8.92% ADE and 17.56% FDE improvement over "Social LSTM."
  • Crowded Scene Performance: The model showed particular strength in crowded scenarios (ETH, Hotel, UCY), where human-human interactions are more complex and critical. It reduced ADE by up to 16.95% and FDE by 18.92% compared to "Social LSTM" in the UCY dataset.
  • Robustness to Parameter Choice: Experiments with different values of $\sigma$ (controlling the Gaussian kernel size for personal space) showed that the model achieves optimal performance within a reasonable range (e.g., $\sigma=8$ for ADE, $\sigma=4$ for FDE), demonstrating its sensitivity to interaction dynamics.
  • Computational Efficiency: The model achieves a quick inference speed of 0.049 seconds per step on a single NVIDIA 3090 GPU, making it suitable for real-time applications.

Limitations and Future Work

  • Special Situations: The authors acknowledge that the model might struggle with highly specific or "emergency" situations like sudden turns or stops, which are not explicitly modeled.
  • Future Work: The paper suggests extending the approach to more diverse scenes and enhancing the "Interaction Module" to better predict these emergent behaviors or sudden changes in pedestrian movement.

Summary updated

Abstract

Predicting the trajectory of pedestrians in crowd scenarios is indispensable in self-driving or autonomous mobile robot field because estimating the future locations of pedestrians around is beneficial for policy decision to avoid collision. It is a challenging issue because humans have different walking motions, and the interactions between humans and objects in the current environment, especially between humans themselves, are complex. Previous researchers focused on how to model human–human interactions but neglected the relative importance of interactions. To address this issue, a novel mechanism based on correntropy is introduced. The proposed mechanism not only can measure the relative importance of human–human interactions but also can build personal space for each pedestrian. An interaction module, including this data-driven mechanism, is further proposed. In the proposed module, the data-driven mechanism can effectively extract the feature representations of dynamic human–human interactions in the scene and calculate the corresponding weights to represent the importance of different interactions. To share such social messages among pedestrians, an interaction-aware architecture based on long short-term memory network for trajectory prediction is designed. Experiments are conducted on two public datasets. Experimental results demonstrate that our model can achieve better performance than several latest methods with good performance.
BibTeX
@article{yang2024ia,
  author = {Jing Yang and Yuehai Chen and Shaoyi Du and Badong Chen and José C. Príncipe},
  title = {{IA-LSTM: Interaction-Aware LSTM for Pedestrian Trajectory Prediction}},
  journal = {IEEE Trans. Cybernetics},
  year = {2024},
  url = {https://openreview.net/forum?id=LTGOWvpr4c},
}
Figure