Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks

Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín, Ian D. Reid, Hamid Rezatofighi, Silvio Savarese

TL;DR

Social-BiGAT is a generative framework that predicts how pedestrians move by combining Graph Attention Networks (GAT) for social modeling with a Bicycle-GAN architecture for diverse, multimodal outputs. By enforcing a reversible mapping between latent noise and trajectories, the model generates multiple realistic future paths that respect both social etiquette and physical scene constraints.

The Challenge of Pedestrian Prediction

Predicting where a person will walk is surprisingly difficult for AI. Humans don't move in a vacuum; our paths are shaped by social interactions (avoiding a collision with a stranger or walking with a friend), scene context (staying on the sidewalk rather than walking through a building), and multimodality.

Conceptual overview of social interaction modeling where pedestrians are nodes in a graph with attention-weighted edges.

Multimodality is a significant hurdle: given the same past movement, a person could turn left, right, or stop. Deterministic models often predict a single "average" path that might collide with obstacles. Early generative models like Social GAN often suffer from "mode collapse," where the random noise meant to drive diversity is ignored, leading to nearly identical predictions.

The Social-BiGAT Architecture

Social-BiGAT (Socially-aware Bicycle-GAN with Graph Attention Networks) addresses these issues through three primary pillars:

  1. Graph Attention Networks (GAT): Pedestrians are treated as nodes in a graph. The GAT learns to assign different "importance" weights to neighbors, allowing the model to prioritize a person directly in the path over someone walking far behind.
  2. Physical Attention: A CNN (VGG) processes the scene image, and soft attention highlights relevant physical features like obstacles or pathways.
  3. Bicycle-GAN Framework: To ensure true multimodality, the model includes a Latent Encoder that learns to map output trajectories back to the original noise vector $z$. This reversible mapping forces the generator to utilize the noise vector, ensuring distinct $z$ values result in distinct, meaningful behaviors.
The full Social-BiGAT architecture, detailing the Generator (G), Local and Global Discriminators (D), and the Latent Encoder (L).

The system is trained with a combination of adversarial losses (using both local and global discriminators), L2 trajectory loss, and a KL-divergence loss to maintain a well-behaved latent space.

Datasets and Evaluation

The model was evaluated on standard pedestrian trajectory benchmarks:

  • ETH: Includes scenes like "ETH" and "HOTEL."
  • UCY: Includes scenes like "UNIV," "ZARA1," and "ZARA2."

These datasets provide real-world coordinates of pedestrians interacting in diverse environments, from busy university walkways to narrow streets.

Key Insights and Results

  • Effective Multimodality: By adopting the Bicycle-GAN approach, the model learns different modes of behavior (e.g., aggressive overtaking vs. cautious avoidance) rather than just adding jitter to a single path.
  • Social Intelligence: The GAT allows the model to recognize that social influence isn't just about proximity; a distant person walking toward you is more relevant than a stationary person nearby.
  • Performance: Social-BiGAT outperformed baselines like S-GAN and Sophie, achieving an Average Displacement Error (ADE) of 0.48m and a Final Displacement Error (FDE) of 1.00m. It showed significant gains in complex scenes like "Hotel," with a 0.15m improvement in FDE over previous state-of-the-art models.
  • Sample Efficiency: The model is robust even with few samples ($K=1$ or $K=5$), producing high-quality distributions from the start.

Limitations and Future Work

While Social-BiGAT sets a new standard, challenges remain:

  • Complexity: The use of GATs and dual discriminators increases computational overhead, which may impact real-time deployment on edge devices like delivery robots.
  • Long-Term Planning: Current predictions span roughly 5 seconds. Future work could incorporate off-screen destinations for longer-term planning.
  • Semantic Understanding: Explicitly labeling scene elements (e.g., "grass" vs. "sidewalk") via semantic segmentation could further refine physical constraints.

Summary updated

Abstract

Predicting the future trajectories of multiple interacting pedestrians in a scene has become an increasingly important problem for many different applications ranging from control of autonomous vehicles and social robots to security and surveillance. This problem is compounded by the presence of social interactions between humans and their physical interactions with the scene. While the existing literature has explored some of these cues, they mainly ignored the multimodal nature of each human's future trajectory which is noticeably influenced by the intricate social interactions. In this paper, we present Social-BiGAT, a graph-based generative adversarial network that generates realistic, multimodal trajectory predictions for multiple pedestrians in a scene. Our method is based on a graph attention network (GAT) that learns feature representations that encode the social interactions between humans in the scene, and a recurrent encoder-decoder architecture that is trained adversarially to predict, based on the features, the humans' paths. We explicitly account for the multimodal nature of the prediction problem by forming a reversible transformation between each scene and its latent noise vector, as in Bicycle-GAN. We show that our framework achieves state-of-the-art performance comparing it to several baselines on existing trajectory forecasting benchmarks.
BibTeX
@inproceedings{kosaraju2019social,
  author = {Vineet Kosaraju and Amir Sadeghian and Roberto Martín-Martín and Ian D. Reid and Hamid Rezatofighi and Silvio Savarese},
  title = {{Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=hhbLry3UspR},
}
Figure