Local-Global Information Interaction Debiasing for Dynamic Scene Graph Generation

Lianli GaoarXiv 2023 · DOI · arXiv

TL;DR

This paper tackles the challenge of "long-tail" data distributions in dynamic scene graph generation (DynSGG), where rare relationships are often missed. They introduce DynSGG-MTL, a novel multi-task learning framework that leverages both local spatial-temporal cues and global long-temporal human actions to improve the prediction of these underrepresented relationships, leading to more accurate scene graphs.

What Problem Does This Paper Address?

Dynamic scene graph generation (DynSGG) aims to represent the evolving relationships between objects and people in videos. This is crucial for understanding complex video content and has applications in video captioning and question answering. However, existing DynSGG models struggle with the "long-tail" distribution of relationship predicates in datasets. This means that while common relationships (e.g., "holding") are predicted well, rare or less frequent ones (e.g., "writing on") are often overlooked. This bias stems from models focusing too heavily on local, frame-by-frame information and failing to capture broader temporal consistencies, leading to inaccurate predictions for tail predicates.

Methods

The proposed framework, DynSGG-MTL, addresses the long-tail problem through a multi-task learning approach that integrates local and global information.

  • Local Information Interaction: The model uses a Spatial-Temporal Transformer (STT) encoder-decoder architecture. Crucially, it incorporates Cross-Attention mechanisms to encode visual, semantic, and frame features. This allows the model to better understand the spatial context within individual frames, mitigating spatial biases.
  • Global Information Interaction: To combat temporal biases and improve tail predicate prediction, the framework leverages long-temporal human actions. The Action Genome dataset provides annotations derived from human actions, allowing the model to learn from these broader temporal patterns. This global context helps enforce temporal consistency across multiple frames.
  • Multi-task Learning: The model is trained on both the main task of dynamic scene graph generation and an auxiliary task of action recognition. This multi-task setup, guided by the long-temporal human actions, helps the model generate scene graphs that adhere to global constraints and better learn tail predicates.
  • Loss Function: A combination of Binary Cross-Entropy (BCE) loss and Focal loss is used for the main task. Focal loss is specifically employed to give more weight to tail predicates, further addressing the long-tail issue.

Datasets

  • Action Genome (AG) dataset: Used for training and evaluation. This dataset provides frame-level scene graph labels derived from human actions in the Charades dataset. It features a long-tailed distribution of predicates and includes annotations for multiple spatial and contact relationships.

Key Insights and Ideas

  • The long-tail problem is a significant bottleneck in DynSGG: The paper clearly articulates how the skewed distribution of relationship predicates leads to poor performance on rare but important interactions.
  • Local and Global information are complementary: The research highlights that relying solely on local, frame-level information is insufficient. Integrating global, long-temporal action information is crucial for achieving robust and unbiased scene graph generation.
  • Cross-Attention is key for local context: The use of Cross-Attention effectively captures the spatial context within frames, helping to disambiguate similar predicates.
  • Auxiliary action recognition task aids debiasing: Training on an auxiliary action recognition task, informed by long-temporal human actions, provides a strong supervisory signal for learning global temporal consistency and improving tail predicate prediction.

Key Results

  • DynSGG-MTL significantly alleviates the long-tail problem in dynamic scene graph generation.
  • The framework achieves state-of-the-art performance on the Action Genome dataset, outperforming previous methods like STTran.
  • Ablation studies confirm the effectiveness of both local (Cross-Attention) and global (long-temporal human actions) information integration. For instance, combining local and global information led to the best performance.
  • Qualitative results show that the proposed model generates more fine-grained relationships compared to the baseline.

Limitations and Future Work

  • Batch Size 1 for Training: The paper mentions that a batch size of 1 was used for training, which can sometimes lead to unstable training. While their results are still strong, exploring larger batch sizes or techniques to mitigate instability could be beneficial.
  • Further Exploration of Global Information: While the paper effectively uses long-temporal human actions, future work could explore other forms of global context or more sophisticated ways to integrate temporal information.
  • Generalization to Other Datasets: While the Action Genome dataset is used, evaluating the framework's generalization capabilities on other dynamic scene graph datasets would be valuable.

Summary updated

Abstract

The task of dynamic scene graph generation (DynSGG) aims to generate scene graphs for given videos, which involves modeling the spatial-temporal information in the video. However, due to the long-tailed distribution of samples in the dataset, previous DynSGG models fail to predict the tail predicates. We argue that this phenomenon is due to previous methods that only pay attention to the local spatial-temporal information and neglect the consistency of multiple frames. To solve this problem, we propose a novel DynSGG model based on multi-task learning, DynSGG-MTL, which introduces the local interaction information and global human-action interaction information. The interaction between objects and frame features makes the model more fully understand the visual context of the single image. Long-temporal human actions supervise the model to generate multiple scene graphs that conform to the global constraints and avoid the model being unable to learn the tail predicates. Extensive experiments on Action Genome dataset demonstrate the efficacy of our proposed framework, which not only improves the dynamic scene graph generation but also alleviates the long-tail problem.
BibTeX
@misc{gao2023local,
  author = {Lianli Gao},
  title = {{Local-Global Information Interaction Debiasing for Dynamic Scene Graph Generation}},
  year = {2023},
  eprint = {2308.05274},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.05274},
}
Figure