TL;DR
CAPSGG introduces a novel framework for Zero-shot Scene Graph Generation (SGG) by leveraging Capsule Networks to model "equivariant" features. By treating unseen relationships as transformations of seen ones in a high-dimensional feature space, the model significantly outperforms existing methods—improving zero-shot performance on the SGCls task by over 132% compared to previous state-of-the-art.
What Problem Does This Paper Address?
Scene Graph Generation (SGG) involves identifying objects in an image and the relationships (predicates) between them, typically represented as triplets like <dog, holding, frisbee>. A major hurdle in this field is the Zero-shot challenge: models often struggle to recognize relationship triplets they did not encounter during training, even if they have seen the individual objects and the predicate separately in different contexts.
The authors identify a "Transformation Gap" as the primary obstacle. Traditional models strive for invariance—the goal of recognizing a predicate like "holding" regardless of whether the subject is a person or a dog. However, the visual manifestation of a relationship changes significantly based on the context, such as the posture of the subjects or their spatial configuration. To bridge this gap, the authors argue that a model needs equivariance: the ability to understand and encode how a feature transforms across different contexts rather than simply ignoring those changes.
Methods: The CAPSGG Framework
The authors propose CAPSGG (Capsule Scene Graph Generation), a framework designed to capture these equivariant properties through a multi-modal approach and specialized network architectures.
1. Three-Stream Pipeline
Instead of relying solely on visual data, CAPSGG extracts features from three distinct modalities to provide a richer context for relationship prediction:
- Visual Stream: Extracts image features of the individual objects and their union area using a backbone network.
- Semantic Stream: Utilizes word embeddings (e.g., GloVe) of the object labels to provide linguistic context.
- Spatial Stream: Encodes the relative bounding box coordinates and geometry of the object pairs.
2. Capsule Feature Encoder (CFE)
The core of the architecture is the Capsule Feature Encoder. Unlike standard neural networks that use scalar neurons, Capsule Networks use vectors (capsules). The length of these vectors represents the probability of a feature's presence, while their orientation encodes "instantiation parameters" such as pose, scale, or transformation. This vector-based representation allows the model to mathematically map the transformations between different contexts of the same predicate.
3. Routing Capsule Layer
To aggregate low-level primary capsules into high-level predicate capsules, the model employs a Self-Attention routing mechanism. This approach is significantly more computationally efficient than the iterative "dynamic routing" used in early Capsule Network implementations, making it more suitable for complex SGG tasks.
4. GroupLoss
To address the "long-tail" distribution problem—where common predicates like "on" or "in" dominate the dataset while others are rare—the authors introduced GroupLoss. This loss function groups identical predicates together during training to balance the learning of invariant and equivariant features, preventing the model from becoming over-biased toward high-frequency relationships.
Datasets and Evaluation
The model was evaluated on the Visual Genome (VG) dataset, specifically the standard VG150 split containing 150 object categories and 50 relation categories. The primary evaluation metric is Zero-Shot Recall (zR@K), which specifically measures the model's ability to generalize to relationship triplets that were absent from the training set.
Key Insights
- Equivariance vs. Invariance: In zero-shot scenarios, the "transformation" between a seen triplet (e.g., boy-holding-frisbee) and an unseen one (e.g., dog-holding-frisbee) contains vital information. Capsules allow the model to map these transformations explicitly.
- Semantic Cues: While some research suggests text features might introduce bias, this study finds that semantic embeddings are essential for providing the conceptual "bridge" that helps the model generalize to new object-predicate combinations.
- Metric Inversion: The authors noted a phenomenon where over-emphasizing negative samples (via a hyperparameter $\lambda$) can lead to "metric inversion," where the model's performance without graph constraints degrades because the distinction between positive and negative classes collapses.
Results and Performance
CAPSGG achieved substantial improvements over previous state-of-the-art (SOTA) models, such as T-CAR:
- SGCls (Scene Graph Classification): Improved zero-shot recall (zR@50) from 12.4 to 28.8, representing a 132.26% increase.
- PredCls (Predicate Classification): Improved zR@50 from 39.8 to 51.7.
- SGDet (Scene Graph Detection): Improved zR@50 from 6.2 to 12.5.
These results demonstrate the robustness of the capsule-based approach across different SGG sub-tasks, both with and without "graph constraints."
Limitations and Future Work
Despite its success, the model has limitations. It is sensitive to the $\lambda$ hyperparameter in GroupLoss; incorrect tuning can trigger the aforementioned metric inversion. Additionally, while Self-Attention routing improves speed, Capsule Networks remain more computationally intensive than standard CNNs. Future research may focus on identifying equivariant features for broader visual reasoning tasks beyond scene graphs.
Summary updated