Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancement

Yuxuan Wang, Xiaoyuan LiuEMNLP 2024 · DOI

Introduction

Researchers have introduced a training-free methodology to enhance Scene Graph Generation (SGG) by leveraging pretrained Vision-Language Models (VLMs). By implementing a novel "Lagrange-Multiplier (LM) Estimation" technique, the researchers mathematically adjust VLM predictions to correct internal biases. This approach significantly improves the recognition of rare and previously unseen object relationships without the need for computationally expensive fine-tuning.

The Long-Tail Challenge in Scene Graph Generation

Scene Graph Generation involves transforming an image into a structured graph where nodes represent objects and edges represent predicates (relationships). A primary obstacle in SGG is the long-tail problem: datasets typically contain abundant examples of common predicates (e.g., "on," "near") but very few for rare ones (e.g., "carrying," "riding").

The long-tail distribution of predicates in SGG datasets and the potential of VLMs to address unseen triplets.

Standard models trained from scratch often fail to generalize to these rare "triplets." While pretrained VLMs like ViLT or Oscar possess broad linguistic knowledge, they suffer from their own "predicate bias," favoring terms frequent in their pretraining data. This misalignment between a VLM's internal distribution and the specific SGG dataset distribution often leads to biased predictions.

Methodology: Training-Free VLM Integration

The proposed framework combines a task-specific SGG model with a zero-shot VLM in a training-free manner, avoiding the high costs of fine-tuning massive models.

The proposed framework integrating zero-shot VLMs with task-specific SGG models using Lagrange-Multiplier Estimation and certainty-aware ensembling.

1. Lagrange-Multiplier (LM) Estimation

Because the pretraining data for many VLMs is proprietary, its exact predicate distribution is unknown. The authors utilize Lagrange-Multiplier Estimation to solve a constrained optimization problem, mathematically estimating the "hidden" distribution the VLM expects based on its performance on a small validation set.

2. Post-hoc Logit Adjustment

With the estimated distribution, a "nudge" is applied to the VLM’s output scores (logits). This debiasing step prevents the model from over-predicting common words, allowing rare predicates to be correctly identified.

3. Certainty-aware Ensemble

A dynamic weighting system evaluates the "certainty" of both the SGG model and the VLM for specific image regions. The system blends their predictions, relying more on the VLM for rare or unseen relationships while trusting the SGG model for standard predicates.

Experimental Results

The study primarily utilized the Visual Genome (VG) dataset, which includes over 108,000 images with 150 object and 50 predicate categories. The method achieved state-of-the-art performance, particularly in mean Recall (mR@K), a metric that weights all classes equally regardless of frequency.

Key findings include:

  • Performance Gains: Adding the debiasing method to a ViLT baseline improved mRecall@100 by 2.0 points in the Predicate Classification task.
  • Unseen Triplets: The debiased VLM significantly outperformed fine-tuned SGG models on triplets never encountered during training, proving that the necessary knowledge was present but previously suppressed by bias.
  • Versatility: The approach showed consistent improvements across different VLM architectures (ViLT, Oscar) and SGG baselines (e.g., PENET).

Discussion and Future Work

The research demonstrates that VLMs act as "latent experts" for rare relationships. Their struggle with "background" classes (no relationship) is mitigated by relying on the task-specific SGG model for detection while using the VLM for semantic identification.

A current limitation is the inference cost, as VLMs are computationally intensive during testing. Future research may focus on optimizing VLM architectures for faster pair-wise processing or extending this debiasing logic to larger models like GPT-4V or LLaVA to further scale performance.

Summary updated

BibTeX
@inproceedings{wang2024predicate,
  author = {Yuxuan Wang and Xiaoyuan Liu},
  title = {{Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancement}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.97},
}
Figure