Object-Centric Representation Learning for Video Question Answering

Long Hoang Dang, Thao Minh Le, Vuong Le, Truyen Tran

Abstract

Video question answering (Video QA) presents a powerful testbed for human-like intelligent behaviors. The task demands new capabilities to integrate video processing, language understanding, binding abstract linguistic concepts to concrete visual artifacts, and deliberative reasoning over spacetime. Neural networks offer a promising approach to reach this potential through learning from examples rather than handcrafting features and rules. However, neural networks are predominantly feature-based - they map data to unstructured vectorial representation and thus can fall into the trap of exploiting shortcuts through surface statistics instead of true systematic reasoning seen in symbolic systems. To tackle this issue, we advocate for object-centric representation as a basis for constructing spatio-temporal structures from videos, essentially bridging the semantic gap between low-level pattern recognition and high-level symbolic algebra. To this end, we propose a new query-guided representation framework to turn a video into an evolving relational graph of objects, whose features and interactions are dynamically and conditionally inferred. The object lives are then summarized into resumes, lending naturally for deliberative relational reasoning that produces an answer to the query. The framework is evaluated on major Video QA datasets, demonstrating clear benefits of the object-centric approach to video reasoning.

Introduction

Most current AI models for video analysis rely on unstructured mathematical vectors, which often leads to "shortcuts"—relying on surface-level statistics (e.g., associating a ball and a net with "basketball") rather than understanding the scene's logic. Humans, however, do not perceive videos as pixel grids; we see objects with unique lives that evolve over time. The challenge lies in bridging the "semantic gap" between low-level pattern recognition and high-level symbolic reasoning.

An example of a Video QA task where the model must track objects and their state changes (rotation, motion) over time to answer correctly.

To address this, we propose an object-centric approach that structures video data into entities and relationships. By treating objects as "living" constructs with a "lifetime," the model mimics human cognitive development, identifying object permanence and cohesion.

Methods

The proposed Object-Centric Representation Learning (OCRL) framework follows a sophisticated pipeline to transform raw video into a format suitable for reasoning:

  1. Tubelet Extraction: Using Faster R-CNN and DeepSort, the model identifies objects and tracks them across frames, creating "tubelets"—3D structures representing an object’s path through space-time.
  2. Position-Specific Gating: A multiplicative gating mechanism combines appearance ("what") with coordinates and relative area ("where"). This allows the model to distinguish between stationary objects and those in motion.
  3. Query-Conditioned Evolving Graphs: The video is divided into short clips. For each, the model builds a graph where objects are nodes. Crucially, the connections (edges) are influenced by the user's question. If a query mentions a "red ball," the model prioritizes that object and its neighbors. A "contextual object" (global frame features) is also included to recover information missed by the detector.
  4. Object Résumés: A BiLSTM summarizes an object's entire history—appearance, movements, and interactions—into a "résumé." This distills complex video into structured data.
  5. Reasoning Engine: These résumés are fed into a general-purpose reasoning engine (such as MACNet or LOGNet) which "reads" the résumés to compute the final answer.

Evaluation and Results

The model was evaluated on three major benchmarks: MSVD-QA (50k questions), MSRVTT-QA (243k questions), and SVQA (Synthetic Video Question Answering). SVQA serves as a "stress test" for multi-step logic, featuring questions like "How many small balls have the same action type as the big black ball?"

OCRL achieved new state-of-the-art (SOTA) results across the board:

  • MSVD-QA: Reached 38.2% accuracy, outperforming previous bests by 2.1 points.
  • MSRVTT-QA: Reached 36.0% accuracy.
  • SVQA: The model reached 79.5% accuracy, a significant jump over non-object-centric methods, proving that object-level tracking is essential for complex logic.

Conclusion

By treating objects as entities with a "lifetime," the OCRL framework effectively bridges the gap between sensory patterns and functional reasoning. While performance remains tied to the quality of the underlying object detector, the use of object résumés allows for a "plug-and-play" approach with various reasoning engines. Future work will explore scaling these representations for longer, more complex videos and further disentangling sensory input from high-level logic.

Summary updated

Abstract

Video question answering (Video QA) presents a powerful testbed for human-like intelligent behaviors. The task demands new capabilities to integrate video processing, language understanding, binding abstract linguistic concepts to concrete visual artifacts, and deliberative reasoning over spacetime. Neural networks offer a promising approach to reach this potential through learning from examples rather than handcrafting features and rules. However, neural networks are predominantly feature-based - they map data to unstructured vectorial representation and thus can fall into the trap of exploiting shortcuts through surface statistics instead of true systematic reasoning seen in symbolic systems. To tackle this issue, we advocate for object-centric representation as a basis for constructing spatio-temporal structures from videos, essentially bridging the semantic gap between low-level pattern recognition and high-level symbolic algebra. To this end, we propose a new query-guided representation framework to turn a video into an evolving relational graph of objects, whose features and interactions are dynamically and conditionally inferred. The object lives are then summarized into resumes, lending naturally for deliberative relational reasoning that produces an answer to the query. The framework is evaluated on major Video QA datasets, demonstrating clear benefits of the object-centric approach to video reasoning.
BibTeX
@inproceedings{dang2021object,
  author = {Long Hoang Dang and Thao Minh Le and Vuong Le and Truyen Tran},
  title = {{Object-Centric Representation Learning for Video Question Answering}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=G4qq6uHxQL},
}
Figure