DDS: Decoupled Dynamic Scene-Graph Generation Network

A. S. M. Iftekhar, Raphael Ruschel, Satish Kumar, Suya You, B. S. Manjunath

Abstract

Scene-graph generation involves creating a structural representation of the relationships between objects in a scene by predicting subject-object-relation triplets from input data. Existing methods show poor performance in detecting triplets outside of a predefined set, primarily due to their reliance on dependent feature learning. To address this issue, we propose DDS -- a decoupled dynamic scene-graph generation network -- that consists of two independent branches that can disentangle extracted features. The key innovation of the current paper is the decoupling of the features representing the relationships from those of the objects, which enables the detection of novel object-relationship combinations. The DDS model is evaluated on three datasets and outperforms previous methods by a significant margin, especially in detecting previously unseen triplets.

Introduction

Dynamic Scene-Graph (DSG) generation aims to map out a video by identifying triplets of $\langle subject, object, relationship \rangle$ for every frame. While current models are effective at recognizing common scenes, they often fail when encountering "unseen triplets"—combinations of objects and actions that were never paired together during training, even if the individual components are known to the model.

Conceptual overview of decoupling for unseen triplet detection through concept transfer.

This failure typically stems from dependent feature learning, where models use the same feature representations for both objects and relationships. This "tangling" causes the model to associate specific actions with specific objects (e.g., learning "ride" only in the context of a "person"). Consequently, when presented with a "dog" in a "riding" posture, the features do not align. DDS (Decoupled Dynamic Scene-Graph) addresses this by decoupling feature learning into two independent transformer-based branches, allowing for the transfer of learned concepts to novel combinations.

The DDS Framework

The core innovation of DDS is a one-stage network that utilizes a two-branch architecture to disentangle object and relationship features, enabling better generalization to rare or unusual scenarios.

The DDS architecture featuring independent object and relation branches with spatio-temporal decoders.

Two-Branch Architecture

Instead of a single pipeline, DDS splits processing into an Object Branch and a Relation Branch. Each branch is equipped with its own transformer-based encoder and spatio-temporal decoder. This separation allows the model to use two distinct sets of learnable queries ($oQ_t$ and $rQ_t$), enabling the object branch to focus on entity appearance while the relation branch targets spatial interaction regions.

Spatio-Temporal Decoding

Each branch employs a two-stage decoding process to maintain consistency and context:

  1. Temporal Decoder: Uses cross-attention to pull information from the previous frame's embeddings ($oE_{t-1}$ and $rE_{t-1}$), ensuring temporal consistency across the video sequence.
  2. Spatial Decoder: Enriches these features with the current frame's spatial context.

Unlike many previous methods, DDS is a one-stage detector; it learns to detect bounding boxes and relationships simultaneously from scratch without relying on a pre-trained, off-the-shelf object detector.

Experimental Evaluation

The researchers evaluated DDS on three major benchmarks to test its robustness and zero-shot capabilities:

  • Action Genome (AG): A large-scale video dataset. The authors introduced a new compositional split specifically to test zero-shot performance, where 80 relationship triplets in the test set are entirely absent from the training set.
  • HICO-Det: A benchmark for static human-object interaction.
  • UnRel: A dataset designed for "unusual" relationships (e.g., "elephant riding bicycle") to test the model's generalization limits.

Key Results and Insights

DDS significantly outperformed existing methods across all benchmarks, particularly in zero-shot scenarios:

  • Compositional Generalization: On the Action Genome compositional split, DDS achieved a Recall@50 of 18.2%, a massive jump from the 4.4% achieved by the previous state-of-the-art (STTran). This represents a substantial improvement in the model's ability to handle unseen triplets.
  • Robustness to Unusual Scenes: On the UnRel dataset, DDS outperformed existing models by 33% in mean Average Precision (mAP), demonstrating its ability to generalize beyond common training patterns.
  • Spatial Specialization: By decoupling the branches, the model naturally learns to look at different parts of the image. The relation branch focuses on the "union box" (the area encompassing both subjects), which contains the most discriminative cues for interactions.
  • Temporal Consistency: The inclusion of the temporal decoder improved performance by approximately 3% recall, proving that "memory" of previous frames is vital for accurate relationship prediction in video.

Conclusion

DDS represents a significant advancement in video understanding by solving the bottleneck of dependent feature learning. By decoupling object and relationship features, the network achieves up to a 24x improvement in detecting unseen triplets compared to previous models. While a performance gap remains between seen and unseen triplets, the decoupled approach provides a robust foundation for future work in generalized dynamic scene-graph generation.

Summary updated

Abstract

Scene-graph generation involves creating a structural representation of the relationships between objects in a scene by predicting subject-object-relation triplets from input data. Existing methods show poor performance in detecting triplets outside of a predefined set, primarily due to their reliance on dependent feature learning. To address this issue, we propose DDS -- a decoupled dynamic scene-graph generation network -- that consists of two independent branches that can disentangle extracted features. The key innovation of the current paper is the decoupling of the features representing the relationships from those of the objects, which enables the detection of novel object-relationship combinations. The DDS model is evaluated on three datasets and outperforms previous methods by a significant margin, especially in detecting previously unseen triplets.
BibTeX
@inproceedings{iftekhar2025dds,
  author = {A. S. M. Iftekhar and Raphael Ruschel and Satish Kumar and Suya You and B. S. Manjunath},
  title = {{DDS: Decoupled Dynamic Scene-Graph Generation Network}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=qatKvgXYwN},
}
Figure