Object-Centric Representation Learning for Video Question Answering

Long Hoang Dang, Thao Minh Le, Vuong Le, Truyen Tran

TL;DR

Researchers have developed a new framework for Video Question Answering (Video QA) that moves away from analyzing raw pixels and instead treats video as a collection of "living" objects. By tracking objects through time and space to create "object résumés," the model can perform complex, human-like reasoning to answer questions about dynamic interactions and temporal changes.

What Problem Does This Paper Address?

Most current AI models for video analysis are "feature-based." They map video data into unstructured mathematical vectors, which often leads the AI to take "shortcuts"—relying on surface-level statistics (like seeing a ball and a net and guessing "basketball") rather than actually understanding the scene's logic.

The authors argue that humans don't see videos as a sequence of pixel grids; we see objects with unique lives that evolve over time. The challenge is bridging the "semantic gap" between low-level pattern recognition (pixels) and high-level symbolic reasoning (logic). To solve this, the paper proposes an object-centric approach that structures video data into entities and relationships that a reasoning engine can actually manipulate.

Methods

The proposed framework, Object-Centric Representation Learning (OCRL), follows a sophisticated pipeline:

  1. Tubelet Extraction: Using Faster R-CNN and DeepSort, the model identifies objects and tracks them across frames, creating "tubelets"—3D structures of an object’s path through space-time.
  2. Position-Specific Gating: Instead of just looking at what an object looks like, the model uses a multiplicative gating mechanism to combine "what" (appearance) with "where" (coordinates and relative area). This helps the model distinguish between an object staying still and an object in motion.
  3. Query-Conditioned Evolving Graphs: The video is broken into short clips. For each clip, the model builds a graph where objects are nodes. Crucially, the connections (edges) between these objects are influenced by the user's question. If you ask about a "red ball," the model prioritizes the red ball and its neighbors in the graph.
  4. Object Résumés: The model uses a BiLSTM (a type of sequential neural network) to summarize an object's entire history into a "résumé." This résumé captures the object's appearance, its movements, and how it interacted with others.
  5. Reasoning Engine: These résumés are fed into a general-purpose reasoning engine (like MACNet or LOGNet) which "reads" the résumés to find the answer to the question.

Datasets

The model was evaluated on three major benchmarks:

  • MSVD-QA: 50k questions over 2k real-world video clips.
  • MSRVTT-QA: A massive dataset with 243k questions over 10k videos.
  • SVQA (Synthetic Video Question Answering): A dataset specifically designed for multi-step reasoning (e.g., "How many small balls have the same action type as the big black ball?"). This is a "stress test" for logic.

Key Insights and Ideas

  • Objects as "Living" Constructs: By treating objects as entities with a "lifetime," the model mimics human cognitive development, where we learn to identify object permanence and cohesion early on.
  • The Power of the "Résumé": Distilling a complex video into a set of object résumés allows the model to "plug-and-play" with different reasoning engines. It turns a messy video problem into a structured data problem.
  • Context Matters: The inclusion of a "contextual object" (global frame features) helps the model recover information that might be lost if the object detector misses a small or blurry item.
  • Query-Guided Representation: Most models look at the video and then the question. This model uses the question to shape how it perceives the video from the start, focusing its "attention" on relevant interactions.

Key Results

The OCRL framework achieved new state-of-the-art (SOTA) results across the board:

  • MSVD-QA: Reached 38.2% accuracy, outperforming the previous best (HCRN) by 2.1 points.
  • MSRVTT-QA: Reached 36.0% accuracy.
  • SVQA: This is where the model truly shined, reaching 79.5% accuracy. This is a significant jump over previous non-object-centric methods, proving that object-level tracking is essential for complex, multi-step logic.

Limitations and Future Work

The authors note that the system's performance is still somewhat tethered to the quality of the underlying object detector and tracker. If the initial "tubelets" are poorly defined due to heavy occlusion or motion blur, the "résumés" will be incomplete.

Future work could involve scaling the number of tubelets for longer, more complex videos and exploring even more sophisticated ways to disentangle sensory patterns from functional reasoning.

Summary updated

Abstract

Video question answering (Video QA) presents a powerful testbed for human-like intelligent behaviors. The task demands new capabilities to integrate video processing, language understanding, binding abstract linguistic concepts to concrete visual artifacts, and deliberative reasoning over spacetime. Neural networks offer a promising approach to reach this potential through learning from examples rather than handcrafting features and rules. However, neural networks are predominantly feature-based - they map data to unstructured vectorial representation and thus can fall into the trap of exploiting shortcuts through surface statistics instead of true systematic reasoning seen in symbolic systems. To tackle this issue, we advocate for object-centric representation as a basis for constructing spatio-temporal structures from videos, essentially bridging the semantic gap between low-level pattern recognition and high-level symbolic algebra. To this end, we propose a new query-guided representation framework to turn a video into an evolving relational graph of objects, whose features and interactions are dynamically and conditionally inferred. The object lives are then summarized into résumés, lending naturally for deliberative relational reasoning that produces an answer to the query. The framework is evaluated on major Video QA datasets, demonstrating clear benefits of the object-centric approach to video reasoning.
BibTeX
@inproceedings{dang2021object,
  author = {Long Hoang Dang and Thao Minh Le and Vuong Le and Truyen Tran},
  title = {{Object-Centric Representation Learning for Video Question Answering}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=G4qq6uHxQL},
}

Loading related content...

Figure