Online Adaptive Temporal Memory With Certainty Estimation for Human Trajectory Prediction

Manh Huynh, Gita AlaghbandWACV 2023

TL;DR

This paper introduces OAT-Mem, a novel framework that enhances existing human trajectory prediction models by leveraging past pedestrian movements without retraining the original model. It achieves this through an augmented temporal memory module and a certainty-based selector, significantly improving prediction accuracy, especially in dynamic and challenging scenarios.

What Problem Does This Paper Address?

Predicting where pedestrians will move next is crucial for autonomous systems like self-driving cars and robots. However, existing models often struggle in real-world scenarios because pedestrian movements can be highly dynamic and unpredictable, differing significantly from the data they were trained on. While some methods try to adapt models during testing, they are often computationally expensive, require access to the predictor's internal workings, or can even worsen predictions when new movements deviate drastically from past ones. This paper tackles the challenge of improving trajectory prediction accuracy online (during testing) without needing to retrain the predictor or know its architecture, making it more flexible and practical for real-world applications.

Methods

The proposed OAT-Mem framework consists of two main novel components:

  • Augmented Temporal Memory (ATM): Inspired by key-value memory networks, ATM stores past pedestrian trajectory samples as "key-value" pairs. During testing, it encodes observed past trajectories (keys) and their corresponding future trajectories (values). When predicting for a new pedestrian, ATM retrieves the most similar past trajectory from its memory to inform the prediction. Crucially, only the motion decoder of the predictor is updated with this memory information, rather than the entire network, making it more efficient.
  • Certainty-based Selector (CS): This module addresses scenarios where past movements are not good indicators of future ones (e.g., abrupt changes in pedestrian behavior or scene context). CS estimates the confidence of OAT-Mem's prediction compared to the original predictor's output. If OAT-Mem is highly confident, its prediction is used; otherwise, the original predictor's output is favored. This prevents the memory from negatively impacting predictions when it's not relevant.

Datasets

The paper evaluates its framework on two commonly used datasets for pedestrian trajectory prediction from ego-centric views:

  • JAAD [12]: Joint Attention for Autonomous Driving dataset.
  • PIE [19]: Pedestrian Intention Estimation dataset.

No new benchmarks were introduced.

Key Insights and Ideas

  • Leveraging Past, Not Retraining: The core idea is that past pedestrian movements, even from different individuals, can provide valuable temporal context for predicting future movements. The framework cleverly exploits this without the computational burden of retraining the entire predictor.
  • The Danger of Irrelevant Memory: Simply relying on memory can be detrimental when scene dynamics change drastically. The certainty-based selector is a critical innovation that acts as a smart gatekeeper, deciding when to trust the memory-enhanced prediction and when to fall back on the original predictor.
  • Selective Adaptation: Adapting only the motion decoder, rather than the entire predictor, is a key efficiency gain. This allows for faster adaptation and reduces the complexity of the update process.
  • Memory Size Matters, But Not Infinitely: The study on memory size reveals that there's an optimal point. Too small a memory might miss crucial temporal patterns, but too large a memory can introduce noise from less relevant past samples, potentially degrading performance.

Key Results

  • Significant Accuracy Improvement: OAT-Mem, when combined with state-of-the-art predictors like BiTrap, PIEtraj, and PIEfull, consistently achieves lower prediction errors (ADE, CADE, CFDE) across both JAAD and PIE datasets, outperforming the base predictors alone. For instance, OATMem + BiTrap achieved the best results across all metrics.
  • Error Reduction on Challenging Scenarios: The framework demonstrated a notable reduction in prediction errors (up to 27%) for samples with large motions, highlighting its effectiveness in dynamic situations.
  • Efficiency Gains: Compared to a baseline like BAOL, OAT-Mem shows significantly lower inference and adaptation times, making it more suitable for real-time applications. It also requires far fewer trainable parameters.
  • Ablation Study Highlights Component Importance: Removing the certainty-based selector resulted in the highest prediction errors, underscoring its crucial role in handling diverse scenarios.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the framework is flexible, its performance is still dependent on the quality of the initial predictor. The effectiveness of the memory relies on the presence of relevant past trajectories.
  • Future Work: The authors suggest extending the framework to multi-agent trajectory prediction and incorporating it with other predictor architectures. Further exploration into more sophisticated memory management and selection strategies could also be beneficial.

Summary updated

BibTeX
@inproceedings{huynh2023online,
  author = {Manh Huynh and Gita Alaghband},
  title = {{Online Adaptive Temporal Memory With Certainty Estimation for Human Trajectory Prediction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure