Memory-augmented Dynamic Neural Relational Inference

Dong Gong, Zhen Zhang, Qinfeng (Javen) Shi, Anton van den Hengel

Introduction

Understanding and predicting the trajectories of complex interacting systems—ranging from physical particles to human social dynamics—remains a fundamental challenge in machine learning. While models like Neural Relational Inference (NRI) and its dynamic counterpart (DNRI) have made strides in inferring latent interactions, they often struggle with long-range dependencies. This paper introduces MemDNRI, a framework that leverages dual external memory pools to overcome the "wash-out" effect inherent in standard recurrent architectures.

The Challenge of Latent Dynamics

In many real-world scenarios, the underlying relationships between entities are unobserved and change over time. For instance, in a basketball game, player movements are influenced by shifting tactical alignments. Previous state-of-the-art models like DNRI utilize LSTMs to track these changing relations. However, as LSTMs process new information, their internal hidden states tend to lose older, critical context. This limitation hinders the ability to model erratic movements or maintain accuracy over extended prediction horizons.

Proposed Method: MemDNRI

MemDNRI addresses the memory bottleneck by augmenting a Variational Autoencoder (VAE) architecture with two distinct associative memory pools:

  1. RelMem (Relation Memory): Stores information regarding the interactions (edges) between entities. To ensure scalability, this memory is shared across all edges, utilizing a "hitting rate" normalization technique to prevent data overwrites.
  2. EntMem (Entity Memory): Tracks the state and history of individual entities. Each entity is assigned a dedicated memory slot to avoid addressing conflicts.
Overview of the MemDNRI architecture showing the integration of Relation Memory (M_rel) and Entity Memory (M_ent) across time steps.

The model employs asymmetrical addressing, using content-based attention for reading and location-based addressing for writing. These memory modules are integrated into Graph Neural Networks (GNNs), where the GNN handles message passing while the memory provides the necessary long-term context.

Experimental Evaluation

The authors evaluated MemDNRI across three diverse domains: synthetic physics simulations, CMU Motion Capture (MoCap) data, and real-world basketball trajectories.

MemDNRI demonstrated superior performance, achieving a relation estimation accuracy of 0.98 on synthetic data (compared to 0.97 for DNRI). Crucially, in the CMU jumping task—characterized by sudden, non-linear movements—MemDNRI maintained a stable Mean Squared Error (MSE) even as the prediction window expanded, whereas previous models saw their error rates explode as they "forgot" the initial state.

Key Insights: Global Latent Structures

A significant finding of this work is that the external memory acts as a global repository rather than a simple buffer. By visualizing the correlation between read and write operations, the authors observed that the model frequently "recalls" information from the very beginning of a sequence to inform predictions at the end.

Visualization of read-write correlation, demonstrating how the model accesses early sequence information to make late-stage predictions.

This behavior confirms that MemDNRI captures long-term global structures. Furthermore, the separation of Relation and Entity memory allows the model to independently learn interaction patterns versus individual entity states, leading to more robust generalization.

Conclusion and Future Work

MemDNRI represents a significant step forward in modeling dynamic interacting systems by providing a mechanism for long-term dependency tracking. While the current implementation introduces some computational overhead and assumes a fully connected graph, future iterations could explore memory integration into the encoder and application to sparser, hierarchical structures.

Summary updated

Abstract

Dynamic interacting systems are prevalent in vision tasks. These interactions are usually difficult to observe and measure directly, and yet understanding latent interactions is essential for performing inference tasks on dynamic systems like forecasting. Neural relational inference (NRI) techniques are thus introduced to explicitly estimate interpretable relations between the entities in the system for trajectory prediction. However, NRI assumes static relations; thus, dynamic neural relational inference (DNRI) was proposed to handle dynamic relations using LSTM. Unfortunately, the older information will be washed away when the LSTM updates the latent variable as a whole, which is why DNRI struggles with modeling long-term dependences and forecasting long sequences. This motivates us to propose a memory-augmented dynamic neural relational inference method, which maintains two associative memory pools: one for the interactive relations and the other for the individual entities. The two memory pools help retain useful relation features and node features for the estimation in the future steps. Our model dynamically estimates the relations by learning better embeddings and utilizing the long-range information stored in the memory. With the novel memory modules and customized structures, our memory-augmented DNRI can update and access the memory adaptively as required. The memory pools also serve as global latent variables across time to maintain detailed long-term temporal relations readily available for other components to use. Experiments on synthetic and real-world datasets show the effectiveness of the proposed method on modeling dynamic relations and forecasting complex trajectories.
BibTeX
@inproceedings{gong2021memory,
  author = {Dong Gong and Zhen Zhang and Qinfeng (Javen) Shi and Anton van den Hengel},
  title = {{Memory-augmented Dynamic Neural Relational Inference}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01163},
}
Figure