Refine and Redistribute: Multi-Domain Fusion and Dynamic Label Assignment for Unbiased Scene Graph Generation

Yujie Zang, Yaochen Li, Yuan Gao, Yimou Guo, Wenneng Tang, Yanxue Li, Meklit AtlawWACV 2024

TL;DR

This paper introduces RAR, a novel framework for unbiased Scene Graph Generation (SGG). RAR tackles the challenges of imbalanced data distributions and weak predicate representations by employing a multi-domain fusion module for feature refinement and a dynamic label assignment strategy for data redistribution, achieving significant improvements in mean recall.

What Problem Does This Paper Address?

Scene Graph Generation (SGG) is crucial for understanding visual scenes by representing objects and their relationships as a graph. However, existing SGG methods struggle with two major issues:

  1. Imbalanced Data Distributions (Long-Tail Problem): Many predicate categories appear infrequently in training datasets, leading to models that perform poorly on these "tail" classes while over-specializing on common "head" classes. This bias hinders accurate and comprehensive scene understanding.
  2. Weak Predicate Representations: Current approaches often fail to capture the subtle nuances of predicate relationships, making it difficult to distinguish between similar predicates (e.g., "on" vs. "standing on"). This is exacerbated by the inherent ambiguity of visual information and limited annotated data.

These issues lead to suboptimal performance and a biased understanding of visual scenes, limiting the applicability of SGG in downstream tasks like image captioning and visual question answering.

Methods

The proposed RAR framework consists of two main components:

  1. Multi-Domain Fusion (MDF) for Feature Refinement: This module aims to create more robust and discriminative predicate representations by integrating information from multiple domains:
  2. Contextual Domain: Uses a Transformer encoder to capture global semantic information from object and relationship sequences.
  3. Spatial-Frequency Domains: Employs a Local Saliency Adapter (LSA) to focus on predicate-specific regions of interest (ROIs). LSA extracts features by:
  4. Spatial: Using deformable convolutions to handle irregular predicate shapes and incorporating attention maps from sliding windows.
  5. Frequency: Applying a Fast Fourier Transform (FFT) to analyze frequency components (edge details, behavioral tendencies) and filtering out low-frequency components.
  6. These multi-domain features are then fused using self-attention to produce a refined predicate representation.
  7. Dynamic Label Assignment (DLA) for Data Redistribution: This plug-and-play module addresses the long-tail problem through adaptive data balancing:
  8. Dynamic Groups Scheduling: Predicates are dynamically grouped into "head," "body," and "tail" based on their instance counts. Training conditions are adaptively adjusted for each group using stepwise factors.
  9. Siamese Space Augmentation (SSA): A hierarchical auto-encoder generates "siamese" duplicate samples to expand the training data, particularly for tail classes, by mimicking original instances.
  10. Auxiliary-Label Optimization (ALO): Leverages annotated samples (L+) to derive high-quality "auxiliary-labels" that act as a strong regularization constraint, shifting decision boundaries towards more general classes.
  11. Pseudo-Label Discrimination (PLD): Identifies and reabsorbs potentially useful unannotated samples (L-) by assigning them "pseudo-labels." This process uses confidence scores and thresholds to filter redundant samples from head classes and include more candidates from tail classes.
  12. ALO and PLD work in a positive feedback loop, with ALO providing proposals for PLD and PLD offering more training data for ALO.

Datasets

The paper evaluates its approach on two standard datasets for scene graph generation:

  • Visual Genome (VG): A benchmark dataset with 108K images and 2.3M relationship annotations. The experiments use the VG150 split, focusing on the 150 most frequent object classes and 50 predicate classes.
  • GQA: A dataset for scene graph-based visual question answering, containing over 200K images and 113K questions. The experiments use the GQA200 split, focusing on 200 object classes and 100 predicate classes.

Key Insights and Ideas

  • Multi-Domain Fusion is Key: The paper strongly advocates for combining contextual, spatial, and frequency domain information to capture richer and more discriminative predicate features. The frequency domain, in particular, is highlighted for its ability to capture fine-grained details crucial for distinguishing tail classes.
  • Dynamic, Not Static, Balancing: Instead of fixed re-weighting or re-sampling, the DLA strategy dynamically adjusts training conditions and data distributions based on predicate group and performance. This adaptive approach is crucial for effectively handling the long-tail problem without sacrificing performance on head classes.
  • Synergy of Data Augmentation and Labeling: The combination of generating synthetic "siamese" samples (SSA) and creating "auxiliary" and "pseudo" labels (ALO/PLD) provides a comprehensive strategy for both expanding the dataset and improving label quality, leading to more robust training.
  • Deformable Convolutions for Irregularity: The use of deformable convolutions is a clever way to adapt feature extraction to the often irregular shapes of predicate regions in images, which standard convolutional filters might miss.

Key Results

  • Significant Performance Gains: The proposed RAR model achieves substantial improvements, with approximately a 10% increase in mean recall compared to baseline models on both VG and GQA datasets.
  • State-of-the-Art Performance: RAR demonstrates state-of-the-art performance across various SGG tasks (Predicate Classification, Scene Graph Classification, and Scene Graph Detection).
  • Ablation Studies Validate Components: Ablation studies confirm the effectiveness of each component: MDF improves performance by around 0.5% to 1%, and DLA provides a further boost of over 2%.
  • Qualitative Improvements: Visualizations show that the model produces more accurate predictions, with green predicates indicating correct matches and red ones indicating errors.

Limitations and Future Work

  • Potential for Overfitting in Tail Classes: While DLA aims to balance training, the paper acknowledges that the dynamic adjustments are crucial to prevent underfitting of head classes and overfitting of tail classes. The specific hyperparameter tuning (like the stepwise factor $\alpha$) is important for this balance.
  • Semantic Ambiguity in Pseudo-Labels: The paper notes that pseudo-labels derived from unannotated samples might carry semantic ambiguity. The mixed augmentation approach (using $\beta$ to combine L+ and LP) is employed to mitigate this, but it remains a potential area for further refinement.
  • SGCls Performance Drop: A minor decrease in SGCls results was observed, attributed to the static nature of SGCls features which might not fully leverage the refined fusion features from MDF. This suggests that future work could explore how to better integrate refined features for tasks that rely on static object features.
  • Computational Complexity: While the paper claims effective doubling of performance with moderate time investment, the complexity of multi-domain fusion and dynamic label assignment might still be a consideration for real-time applications.

The paper suggests that the proposed framework's ability to dynamically adjust training conditions and data distributions offers a promising direction for tackling the long-tailed problem in SGG and other related vision tasks.

Summary updated

BibTeX
@inproceedings{zang2024refine,
  author = {Yujie Zang and Yaochen Li and Yuan Gao and Yimou Guo and Wenneng Tang and Yanxue Li and Meklit Atlaw},
  title = {{Refine and Redistribute: Multi-Domain Fusion and Dynamic Label Assignment for Unbiased Scene Graph Generation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure