DDS: Decoupled Dynamic Scene-Graph Generation Network

A S. M. Iftekhar, Raphael Ruschel, Satish Kumar, Suya You, B. S. Manjunath

Abstract

Scene-graph generation involves creating a structural representation of the relationships between objects in a scene by predicting subject-object-relation triplets from input data. Existing methods show poor performance in detecting triplets outside of a predefined set, primarily due to their reliance on dependent feature learning. To address this issue we propose DDS– a decoupled dynamic scene-graph generation network- that consists of two independent branches that can disentangle extracted features. The key innovation of the current paper is the decoupling of the features representing the relationships from those of the objects, which enables the detection of novel object-relationship combinations. The DDS model is evaluated on three datasets and outperforms previous methods by a significant margin, especially in detecting previously unseen triplets.

Introduction: The Challenge of Compositionality

In computer vision, a Dynamic Scene-Graph (DSG) is a structured representation of video content where each frame is decomposed into triplets: <subject, relation, object> (e.g., <person, riding, bicycle>).

A significant hurdle in this field is compositionality. Most existing models learn features for these triplets dependently. For instance, if a model only encounters "people riding bikes" and "dogs sleeping on beds" during training, it often fails to recognize a "dog riding a bike" because the concept of "riding" has become entangled with the visual features of a "person."

Concept of decoupling object and relation branches to enable detection of unseen triplets like a dog riding a bicycle.

The authors address this "feature entanglement" by proposing a system that disentangles objects from their relationships, allowing for superior generalization to rare or unusual events.

Methods: The Decoupled Dynamic Scene-Graph (DDS) Network

The core innovation is the DDS network, a one-stage, multi-branch architecture that avoids the bottlenecks of traditional two-stage models relying on pre-trained object detectors.

The DDS architecture featuring independent object and relation branches with spatio-temporal decoders.

Architecture Overview

  1. Two-Branch Design: The network splits into an Object Branch and a Relation Branch. This ensures that features identifying a "bicycle" are distinct from those identifying the act of "riding."
  2. Transformer-Based Spatio-Temporal Decoders: Each branch utilizes a transformer encoder-decoder structure. The Temporal Decoder integrates queries from the current frame with embeddings from the previous frame ($t-1$), enabling the model to track evolving relationships. The Spatial Decoder then refines these queries using the spatial context of the current frame.
  3. Independent Queries: Each branch employs its own set of randomly initialized, learnable queries ($oQ_t$ and $rQ_t$). This prevents "feature bleeding," where the same features are forced to represent both object identity and action.

Experimental Setup and Datasets

The DDS model was evaluated on three major benchmarks:

  • Action Genome (AG): The primary dataset for dynamic scene graphs. The authors introduced a new compositional split for AG to test the detection of 80 relationship triplets never seen during training.
  • HICO-DET: A standard benchmark for static scene-graph (SSG) generation.
  • UnRel: A challenging dataset containing "unusual" relations (e.g., an elephant on a bicycle) to test the limits of generalization.

Key Insights and Results

The primary insight is that forcing the network to attend to different spatial regions for objects versus relations leads to significantly higher accuracy. The object branch focuses on item-specific features, while the relation branch focuses on the interaction space between entities.

Performance Highlights

DDS outperformed existing State-of-the-Art (SOTA) models across all benchmarks, with particularly striking results on "unseen" data:

  • Action Genome (Unseen Triplets): DDS achieved a 4x to 24x improvement over the previous leading model (STTran) in zero-shot scenarios.
  • Action Genome (Supervised): Showed an average recall enhancement of 7% in standard settings.
  • UnRel Dataset: Outperformed existing models by 33% in mean Average Precision (mAP), demonstrating robustness in "weird" visual scenarios.
  • HICO-DET: Improved SOTA by 5% for static scene graphs.

Conclusion and Future Work

DDS represents a major advancement in detecting unseen triplets by leveraging decoupled feature learning. While it significantly narrows the gap, the authors note that detecting novel combinations remains more challenging than familiar ones. Future research will focus on further improving the generalization of the Relation Branch and refining "union box" logic to better capture non-contact interactions in diverse, large-scale datasets.

Summary updated

Abstract

Scene-graph generation involves creating a structural representation of the relationships between objects in a scene by predicting subject-object-relation triplets from input data. Existing methods show poor performance in detecting triplets outside of a predefined set, primarily due to their reliance on dependent feature learning. To address this issue we propose DDS– a decoupled dynamic scene-graph generation network- that consists of two independent branches that can disentangle extracted features. The key innovation of the current paper is the decoupling of the features representing the relationships from those of the objects, which enables the detection of novel object-relationship combinations. The DDS model is evaluated on three datasets and outperforms previous methods by a significant margin, especially in detecting previously unseen triplets.
BibTeX
@inproceedings{iftekhar2025dds,
  author = {A S. M. Iftekhar and Raphael Ruschel and Satish Kumar and Suya You and B. S. Manjunath},
  title = {{DDS: Decoupled Dynamic Scene-Graph Generation Network}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=qatKvgXYwN},
}
Figure