Relational Temporal Modeling with Graph Neural Networks

This direction is split into other directions: graph-based temporal refinement. Its last version stays readable and citable here.

Version 1 of 1139 papers179 conceptsfrozen 2026-10-04

Traditional sequence models like RNNs and LSTMs treat time as a "film strip"—a rigid, linear progression where information flows strictly from one frame to the next. While intuitive, this framing struggles with long-range dependencies, irregular sampling, and the inherent redundancy of high-frequency data.

A growing body of research proposes a shift in perspective: Video and longitudinal data are not just sequences; they are graphs. By representing time steps, video segments, or trajectories as nodes in a graph, researchers can model complex temporal relationships as edges, allowing for direct information exchange between distant but semantically related events.

Breaking the Linear Constraint

The primary limitation of sequential processing is the "bottleneck" of incremental updates. In a standard recurrent architecture, to connect frame 1 to frame 100, the model must pass a hidden state through 98 intermediate steps, often losing signal along the way.

Byoung-Tak Zhang addresses this in Visualizing Semantic Structures of Sequential Data by Learning Temporal Dependencies by learning a weighted adjacency matrix between all frames. This allows the model to bypass the local transition bias. Instead of a chain, the sequence becomes a web where a "throw" action can be directly linked to a "catch" action seconds later, regardless of the "background noise" in between. Similarly, Daniele Zambon and colleagues demonstrate in Autoregressive Models for Sequences of Graphs that graph-aware architectures maintain significantly higher predictive accuracy for complex topologies over long durations compared to traditional moving average baselines.

Handling Irregularity and Redundancy

Real-world temporal data is rarely "clean." Medical records have gaps, and sensors may trigger at irregular intervals. Ziqi Chen argues in Sampling-guided Heterogeneous Graph Neural Network with Temporal Smoothing for Scalable Longitudinal Data Imputation that modeling observations as nodes in a heterogeneous graph eliminates the need for artificial padding or alignment. Temporal dependencies are simply edges with attributes representing the time delta.

Furthermore, sequential models often suffer from "information redundancy"—processing every frame even when nothing changes. Yiwei Wang introduces Time-aware Neighbor Sampling (TNS) in Time-Aware Neighbor Sampling for Temporal Graph Networks to solve this. By learning to skip redundant recent interactions and adaptively expanding the "temporal receptive field," the model captures long-range patterns that would otherwise be obscured by high-frequency bursts of activity.

Scaling via Dilated and Sparse Graphs

If every frame is a node, a fully connected graph becomes computationally explosive ($O(N^2)$). To solve this, researchers use structural priors to prune the graph.

  1. Dilated Connectivity: Dong Wang and Di Hu proposed the Dilated Temporal Graph Reasoning Module (DTGRM) in Temporal Relational Modeling with Self-Supervision for Action Segmentation. Much like dilated convolutions in CNNs, nodes connect to neighbors at exponentially increasing distances. This provides a massive receptive field with minimal connections.
  2. Sparse Sampling: In Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection, Kyle Min uses sparse graph structures to process temporal windows of up to 55 seconds—nearly 30 times longer than the 2-second limit of traditional dense architectures.
  3. Adaptive Construction: Ping Yu and Chunyuan Li developed SA-GCN in Structure-Aware Human-Action Generation, which uses self-attention to dynamically select only the most relevant past frames to build a sparse, task-specific graph.

Virtual and Semantic Bridging

Perhaps the most powerful aspect of the graph perspective is the ability to create "virtual" edges. In Dynamic Virtual Graph Significance Networks for Predicting Influenza, Jie Zhang and Hongyan Wu link the current state to similar historical time points in a latent similarity space. This allows the model to "remember" how a previous outbreak behaved, even if that event occurred years ago.

This concept extends to multi-object tracking and activity recognition. Andrew Bradley and Fabio Cuzzolin demonstrate in A Hybrid Graph Network for Complex Activity Detection in Video how local scene graphs (modeling agent interactions) can be nested within a global temporal graph (modeling activity progression). By treating the entire video as a hierarchical graph, the system can maintain identity consistency and activity context across extended durations that would baffle a standard sequential model.

Papers in this version

Showing 60 of 139 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_relational_temporal_modeling_with_graph__ab5751d4,
  title = {Relational Temporal Modeling with Graph Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relational-temporal-modeling-with-graph-neural-networks/rdr_ab5751d4463f4d4f9808a70b892ccb5f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_relational_temporal_modeling_with_graph__ab5751d4_v1,
  title = {Relational Temporal Modeling with Graph Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relational-temporal-modeling-with-graph-neural-networks/rdr_ab5751d4463f4d4f9808a70b892ccb5f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}