Mitigating Predicate Frequency Bias in Scene Graph Generation

550 concepts267 papers1 proposal

Scene Graph Generation (SGG) aims to convert an image into a structured directed graph where nodes represent objects (e.g., man, bicycle) and edges represent their semantic relationships via predicates (e.g., riding, parked on). While deep learning models excel at identifying objects, they consistently fail to accurately predict relationships.

The root cause is a severe long-tail predicate distribution. In standard benchmarks like Visual Genome, a handful of generic spatial predicates (the "head," such as on, near, and has) appear orders of magnitude more frequently than specific, descriptive predicates (the "tail," such as parked on, standing on, or eating).

Predicate Frequency
▲
│  █
│  █  ◄── Head Predicates (e.g., "on", "near")
│  █      High frequency, low information
│  █
│  █ ▄ ▂  ◄── Tail Predicates (e.g., "parked on", "walking in")
│  ████████▄▄▄▄▄▂▂▂▂▂▂      Low frequency, high information
└────────────────────────► Predicate Classes

When models are trained on this distribution, they learn to maximize global accuracy by defaulting to statistically "safe" head predicates. If a model sees a truck and a street, predicting on is almost always statistically correct, even if parked on is visually true and far more informative. This frequency bias strips scene graphs of the granular semantic detail required for downstream reasoning tasks like Visual Question Answering (VQA) or robotic navigation.


The Mechanisms Driving Predicate Bias

To understand why standard models fail, we must look at how the long-tail distribution distorts the learning process through two primary mechanisms: reporting bias and semantic overlap.

Reporting Bias and Statistical Priors

Human annotators naturally exhibit reporting bias. They rarely label obvious spatial configurations unless they are notable, yet when they do label common configurations, they default to simple terms. This creates a strong statistical co-occurrence prior between object pairs and predicates.

Standard models learn to rely heavily on these object co-occurrence priors rather than grounding their predictions in visual evidence. If the training data dictates that (person, horse) is overwhelmingly paired with riding, the model will predict riding even if the person is merely standing next to the horse. Conversely, for rare combinations or less frequent predicates, the visual features are overridden by the dominant statistical prior of the head classes.

Semantic Overlap and Representation Collapse

Many head and tail predicates share significant semantic and visual overlap. For instance, sitting on, standing on, and parked on are all fine-grained sub-classes of the generic spatial predicate on.

During training, the loss gradients generated by millions of instances of on dwarf the gradients from a few dozen instances of sitting on. As a result, the relational encoder's feature space collapses: the distinct visual features that characterize the tail predicates are mapped directly into the representation space of the dominant head predicate. The model loses the capacity to differentiate the subtle visual nuances that separate a generic relationship from a specific one.


Countering Bias via Causal Inference

One of the most elegant solutions to reporting bias does not alter the training distribution at all; instead, it removes the statistical prior during inference using causal counterfactual reasoning.

In Unbiased Scene Graph Generation From Biased Training, Kaihua Tang et al. framed SGG using causal graphs to isolate the bad effects of context. They identified that the final predicate prediction is driven by two distinct causal paths:

  1. The Direct Visual Path: Visual Features -> Predicate
  2. The Biased Contextual Path: Object Labels -> Statistical Prior -> Predicate
  [ Visual Features ] ───► [ Predicate Prediction ]
                                ▲
                                │
  [  Object Labels  ] ──────────┘
     (Creates the harmful statistical prior)

To eliminate the second path's dominance, the authors utilized Total Indirect Effect (TIE). During inference, they calculate a counterfactual: What would the model predict if it only knew the object labels but had no visual features?

By subtracting this "blind" statistical prediction from the total prediction calculated with the visual features, the model effectively strips away the dataset's reporting bias. This forces the network to rely on the actual visual evidence, immediately elevating the recall of rare tail predicates without requiring retraining or data manipulation. Expanding on multi-level dependencies, Li Liu et al. demonstrated in A Causal Adjustment Module for Debiasing Scene Graph Generation that modeling these causal links across individual objects and specific object pairs prevents the model from collapsing rare interactions into generic predictions.


Imposing Hierarchical Structure on the Feature Space

Another strategy tackles semantic overlap by forcing the model to learn relationships in a coarse-to-fine manner, preventing tail representations from being swallowed by head classes.

In CogTree: Cognition Tree Loss for Unbiased Scene Graph Generation, Jing Yu 0007 et al. organized relationship labels into a hierarchical "cognition tree" derived from model confusion matrices. Instead of treating predicate classification as a flat, single-step multi-class problem, the loss function forces the model to make a series of hierarchical decisions.

                      [ Root ]
                      ╱      ╲
          [ Spatial Rel. ]    [ Action Rel. ]
             ╱        ╲             │
         [ on ]     [ under ]   [ eating ]
          ╱  ╲
 [sitting on] [parked on]

The model first learns to distinguish broad, coarse categories (e.g., distinguishing a spatial relationship from an action relationship). Once it successfully navigates to the correct spatial branch, it is penalized specifically for confusing fine-grained tail predicates (like parked on vs sitting on). This hierarchical structure ensures that tail predicates maintain their own distinct regions within the feature space.

Similarly, Misaki Ohashi et al. proposed in Unbiased Scene Graph Generation using Predicate Similarities to cluster semantically similar predicates into explicit branches. By implementing a knowledge transfer module, their framework allows data-starved tail predicates to "borrow" structural and visual features from their high-frequency head counterparts within the same semantic branch, smoothing the learning curve for rare labels.


Decoupling Representation and Classification

Standard training couples feature extraction and classification into a single end-to-end objective. Because head predicates appear so frequently, the feature extractor learns excellent representations for head classes, but the classifier's decision boundaries become heavily skewed.

To break this dependency, Alakh Desai et al. introduced a decoupled training paradigm in Learning of Visual Relations: The Devil is in the Tails. Their approach splits training into two distinct phases:

  1. Long-Tailed Representation Learning: The model trains normally on the original biased dataset. This allows the relational encoder to learn rich, generalized visual features from the abundant head data.
  2. Balanced Classifier Learning: The weights of the feature encoder are frozen. The classification head is then retrained using Alternating Class-Balanced Sampling (ACBS), which artificially balances the frequency of head and tail predicates in each batch.

Because the encoder is frozen, the model cannot overfit its visual representations to the rare classes. Meanwhile, the classifier is forced to draw fair, balanced decision boundaries between classes, preventing it from defaulting to high-frequency priors.


Curriculum and Collaborative Learning Strategies

Rather than treating all samples equally throughout training, advanced frameworks dynamically adjust the learning focus based on predicate frequency and model capability.

In Learning To Generate Scene Graph from Head to Tail, Chaofan Zheng et al. introduced a curriculum-based re-weighting strategy. The model begins by mastering robust, generalized features from common head predicates. As training progresses, a semantic consistency module gradually shifts the optimization focus toward rare, fine-grained interactions. This ensures the model does not forget foundational spatial concepts while mastering tail nuances.

Complementing this temporal adjustment, Xingning Dong et al. proposed a spatial-collaborative approach in Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation. They utilized a Group Collaborative Learning (GCL) strategy that divides the predicate distribution into distinct frequency groups. Multiple classifier heads collaborate and cross-supervise each other across these groups, balancing the prediction space and significantly boosting mean Recall (mR@K) without sacrificing performance on high-frequency predicates.


Summary of Core Intuitions

The shift from biased baselines to unbiased scene graph generation relies on a few fundamental insights:

Strategy Core Intuition Key Benefit
Causal Inference (Tang et al.) Subtracts the "blind" object-label prior at inference time. Eliminates reporting bias without retraining.
Hierarchical Trees (Yu et al.) Forces coarse-to-fine classification decisions. Prevents tail representations from collapsing into head classes.
Decoupled Training (Desai et al.) Freezes feature representations before balancing the classifier. Retains rich visual features while fixing decision boundaries.
Knowledge Transfer (Gao et al.) Shares visual attributes from head predicates to tail predicates. Enables learning of rare predicates with minimal samples.

By addressing the structural, statistical, and semantic drivers of predicate frequency bias, these methods ensure that scene graphs capture the true visual complexity of the world rather than merely regurgitating dataset statistics.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 341 papers
  2. Version 1 · 2026-10-04 · 199 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_long_tail_predicate_distribution_a8d0e9c5,
  title = {long-tail predicate distribution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/long-tail-predicate-distribution/rdr_a8d0e9c5c34a4e629dda9d6e8723ee86},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_long_tail_predicate_distribution_a8d0e9c5_v2,
  title = {long-tail predicate distribution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/long-tail-predicate-distribution/rdr_a8d0e9c5c34a4e629dda9d6e8723ee86/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake