Multiple Trajectory Prediction of Moving Agents With Memory Augmented Networks

Francesco Marchetti, Federico Becattini, Lorenzo Seidenari, Alberto Del Bimbo

Abstract

Pedestrians and drivers are expected to safely navigate complex urban environments along with several non-cooperating agents. Autonomous vehicles will soon replicate this capability. Each agent acquires a representation of the world from an egocentric perspective and must make decisions ensuring safety for itself and others. This requires predicting motion patterns of observed agents for a far enough future. In this paper, we propose MANTRA, a model that exploits memory-augmented networks to effectively predict multiple trajectories of other agents, observed from an egocentric perspective. Our model stores observations in memory and uses trained controllers to write meaningful pattern encodings and read trajectories that are most likely to occur in the future. We show that our method is able to natively perform multi-modal trajectory prediction, obtaining state-of-the-art results on four datasets. Moreover, thanks to the non-parametric nature of the memory module, we show how once trained, our system can continuously improve by ingesting novel patterns.

Introduction

Predicting the future motion of surrounding agents is a cornerstone of safe autonomous driving. However, this task is plagued by two major challenges: egocentric perception and multimodality. Most vehicles perceive the world from their own moving perspective, requiring them to infer the intent of others from limited viewpoints. Furthermore, the future is inherently uncertain; at an intersection, an agent might turn left, right, or continue straight. Standard neural networks often struggle with this uncertainty, sometimes predicting a "mean" trajectory that is physically impossible (e.g., driving through a building) because it is the mathematical average of divergent paths.

MANTRA (Memory Augmented Neural TRAjectory predictor) addresses these issues by leveraging an explicit associative memory. Instead of relying solely on internal weights to compress information, MANTRA says, "I've seen a situation like this before, and here is what happened."

The MANTRA Architecture

Unlike standard Recurrent Neural Networks (RNNs), MANTRA utilizes an external, addressable memory module to store and recall motion patterns.

Overview of the MANTRA pipeline: from input past trajectory and context map to memory-based multimodal future prediction.

Core Components

  1. Encoders: The system employs dual encoders to process the agent's past trajectory and the "context" (a top-view semantic map of the environment).
  2. Memory Bank: The memory stores triplets consisting of {Past Encoding, Context Encoding, Future Encoding}.
  3. Writing Controller: Acts as a gatekeeper. During training, it calculates the reconstruction error of a sample. If the model cannot accurately predict the future of a seen sample, the controller writes it into memory, ensuring the bank only stores novel or difficult patterns.
  4. Reading Controller: At inference time, the model identifies the Top-K most relevant entries in its memory based on a learned similarity function.
  5. Decoder: The decoder uses retrieved memory samples as "templates," refining them based on current observations to produce final coordinate sequences.

Key Insights

Native Multimodality

By retrieving multiple samples from memory, MANTRA naturally produces distinct hypotheses for an agent's future. This avoids the "averaging" problem of traditional models, allowing the system to simultaneously consider stopping, turning, or accelerating.

Egocentric view of multimodal trajectory prediction for a cyclist, showing multiple potential future paths.

Incremental Learning and Context

Because the memory is non-parametric, the system can improve online without retraining neural weights. New patterns can simply be added to the memory bank. Additionally, incorporating environmental context (the map) as a constraint significantly reduces "off-road" predictions, as the map informs the model where an agent can move, while the trajectory informs how they move.

Experimental Results

MANTRA was evaluated on four major autonomous driving benchmarks: KITTI, Argoverse, Oxford RobotCar, and Cityscapes.

  • State-of-the-Art Performance: MANTRA outperformed existing models like DESIRE and INFER on KITTI and Argoverse across standard metrics, including Average Displacement Error (ADE) and Final Displacement Error (FDE).
  • Real-time Efficiency: The model achieves an inference speed of approximately 65Hz on a modern GPU, making it suitable for real-time autonomous applications.
  • Robustness: In "zero-shot" transfer tests (e.g., training on KITTI and testing on Oxford RobotCar), MANTRA maintained high accuracy despite changes in driving sides and layouts, proving the generalizability of its learned representations.

Conclusion and Future Work

MANTRA demonstrates that an explicit memory of past experiences is a powerful tool for trajectory prediction. While the current model treats agents in isolation, future work will focus on social interaction modeling—accounting for how multiple agents influence each other's paths—and the addition of episodic memory to handle immediate scene-specific contexts.

Summary updated

Abstract

Pedestrians and drivers are expected to safely navigate complex urban environments along with several non cooperating agents. Autonomous vehicles will soon replicate this capability. Each agent acquires a representation of the world from an egocentric perspective and must make decisions ensuring safety for itself and others. This requires to predict motion patterns of observed agents for a far enough future. In this paper we propose MANTRA, a model that exploits memory augmented networks to effectively predict multiple trajectories of other agents, observed from an egocentric perspective. Our model stores observations in memory and uses trained controllers to write meaningful pattern encodings and read trajectories that are most likely to occur in future. We show that our method is able to natively perform multi-modal trajectory prediction obtaining state-of-the art results on four datasets. Moreover, thanks to the non-parametric nature of the memory module, we show how once trained our system can continuously improve by ingesting novel patterns.
BibTeX
@article{marchetti2020multiple,
  author = {Francesco Marchetti and Federico Becattini and Lorenzo Seidenari and Alberto Del Bimbo},
  title = {{Multiple Trajectory Prediction of Moving Agents With Memory Augmented Networks}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2020},
  doi = {10.1109/tpami.2020.3008558},
}
Figure