Spatiotemporal Graph Representations for Video Reasoning

328 concepts226 papers3 proposals

Standard computer vision models treat video as a dense, three-dimensional grid of pixels—two dimensions for space, and one for time. To recognize an action or answer a question about a video, traditional architectures like 3D convolutional networks or space-time Transformers must process every pixel across every frame.

This approach is computationally expensive and fundamentally wasteful. If a video depicts a person washing a tomato, the background kitchen tiles remain static, yet a dense model expends massive compute analyzing them frame after frame. More importantly, dense pixel grids struggle with long-range dependencies and structural logic. If the tomato is temporarily hidden behind a blender, a pixel-level attention mechanism often loses track of it because it lacks an explicit concept of object permanence.

To overcome these limitations, a powerful paradigm has emerged: representing video not as a rigid grid of pixels, but as a dynamic, spatiotemporal graph of interacting entities. By abstracting raw video into structured graphs, models can filter out environmental noise, maintain object identities across occlusions, and perform high-level causal reasoning.


Abstracting Video into a Graph

The core intuition behind this approach is that a video is a sequence of events driven by discrete entities (actors and objects) and their evolving relationships. To capture this structurally, we can map a video into a graph where:

  • Nodes represent detected objects, actors, or localized semantic regions.
  • Edges represent the relationships between these entities across two distinct dimensions: space and time.
[ Frame t ]                 [ Frame t+1 ]

  Actor (Node A) ----------> Actor (Node A')  <-- Temporal Edge (Identity/Motion)
     |                         |
     | Spatial                 | Spatial
     v                         v
  Object (Node B) ---------> Object (Node B') <-- Temporal Edge (State Change)

To construct these graphs, models typically use an object detector to identify candidate regions in each frame. The challenge then lies in wiring these nodes together meaningfully without creating a prohibitively dense, fully connected network.

As Boxiao Pan et al. demonstrate, effective architectures use structural inductive biases to constrain these connections. Spatial edges are established within a single frame, connecting nodes based on geometric proximity or intersection-over-union (IoU) metrics (e.g., linking a hand node to a cup node). Temporal edges are constructed across frames, linking semantically or geometrically similar objects across adjacent time steps to track their trajectories.

By transitioning from pixels to a sparse graph, the model strips away irrelevant visual noise like lighting variations and background textures. As noted by Andrei Liviu Nicolicioiu et al., treating time as an explicit relational axis rather than a simple third spatial dimension allows models to capture the deep temporal logic required to understand complex human-object interactions.


How Information Flows: Relational Message Passing

Once a spatiotemporal graph is constructed, the model reasons over it using Graph Convolutional Networks (GCNs) or Graph Neural Networks (GNNs). Instead of updating a pixel's representation based on its spatial neighbors, a GCN updates an entity's representation based on its relational neighborhood.

Information propagates along the graph's edges through message passing. For example, in an action detection task, an actor node might be physically distant from an object node in early frames, but as they interact, the spatial and temporal edges guide the flow of information.

Pavel Tokmakov et al. showed that treating actors and objects as nodes in a GCN enables the propagation of motion and intent across the entire scene. If an actor begins to lift a box, the actor's motion features flow along the spatial edge to update the box's state representation. Concurrently, the box's historical trajectory flows along temporal edges to inform the actor's current state.

Modeling these explicit relationships yields far more informative visual representations than treating objects as independent entities, a principle thoroughly validated by Yaya Shi et al.. The network learns to classify actions based on structural transformations (e.g., how the distance vector between two nodes changes over time) rather than relying purely on static appearance cues.


Overcoming Occlusion and Noise

A major advantage of spatiotemporal graphs is their resilience to visual disruptions. In dense video models, if an object passes behind an obstacle, its visual features disappear, often causing the model to lose track of its identity or role in the action.

Graphs handle this by decoupling an entity's identity from its immediate visibility. Because temporal edges can be wired across non-consecutive frames or guided by motion patterns, the graph acts as a structural anchor.

For instance, Can Zhang 0001 utilizes sparse motion graphs that link distant video segments sharing similar motion dynamics. If an object is occluded, the long-range temporal edges allow information to bypass the occluded frames entirely, preserving the continuity of the action track. The model maintains a persistent representation of the entity, understanding that it still exists and retains its relational context within the scene even when it cannot be seen.


A Natural Bridge to Language and Reasoning

Beyond improving action recognition, representing videos as spatiotemporal graphs provides a clean interface for high-level cognitive tasks, such as video-grounded dialogue and question answering.

Human language is inherently compositional; we describe the world using subjects, verbs, and objects. A spatiotemporal scene graph naturally decomposes continuous visual data into discrete semantic triplets—(subject, predicate, object)—that evolve over time. This structural alignment makes it significantly easier for language models to parse visual events.

[ Visual Graph ]                              [ Language Model ]

(Person) --[holding]--> (Cup)  == Abstracted ==> "What is the person holding?"
   |                       |     Triplets
   v                       v
(Person) --[drinking]-> (Cup)                   Answer: "A cup."

In video question answering, relying on flat sequences of visual features forces a model to guess answers based on statistical correlations. In contrast, Shijie Geng et al. and Hongcheng Liu et al. demonstrate that feeding structured scene graphs into language models enables multi-step logical reasoning. The language model can traverse the graph's paths to verify causal chains (e.g., confirming that an object was placed on a table before a person left the room).

This compositionality is precisely why benchmarks like AGQA, developed by Ranjay Krishna, use spatiotemporal scene graphs as their underlying foundation to generate intricate questions that rigorously test an AI's causal and relational understanding.

This bridge functions in reverse as well. For video generation, Hao Fei et al. use large language models to first convert a text prompt into a detailed, time-ordered dynamic scene graph. This graph serves as a structural blueprint, guiding diffusion models to generate pixels that adhere to strict physical and relational constraints, resulting in videos with superior temporal consistency.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 232 papers
  2. Version 1 · 2026-10-04 · 431 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_spatio_temporal_relational_modeling_e90a020f,
  title = {spatio-temporal relational modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatio-temporal-relational-modeling/rdr_e90a020f93e64fd3a66d63e7a8de6cf4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_spatio_temporal_relational_modeling_e90a020f_v2,
  title = {spatio-temporal relational modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatio-temporal-relational-modeling/rdr_e90a020f93e64fd3a66d63e7a8de6cf4/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake