Introduction
Scene Graph Generation (SGG) is the task of transforming an image into a structured representation where nodes represent objects (e.g., "man," "bike") and edges represent their semantic relationships (e.g., "man riding bike"). While standard SGG relies on dense, expensive human annotations—where every object requires a bounding box and a defined relationship—Weakly-Supervised SGG (WS-SGG) aims to learn from image-level "ungrounded" scene graphs.
The primary challenge in WS-SGG is the correspondence problem: determining which image regions correspond to specific labels in the provided graph without explicit location data. Previous approaches often treated this as a complex latent variable problem, leading to computationally heavy and unstable training. This paper introduces a surprisingly effective "simple baseline" that outperforms previous state-of-the-art methods by over 30% on the Visual Genome dataset.
Methodology: Decoupled Graph Matching
The core innovation of this work is the decoupling of the correspondence problem from the generation task into a two-stage pipeline.
1. Weakly-Supervised Graph Matching (WSGM)
Instead of training the entire system at once, the authors first solve the matching problem. They map region proposals and label graphs into a shared embedding space using an embedding function $F$.
- Hungarian Algorithm: To ensure global consistency, the Hungarian algorithm is employed to find the optimal one-to-one assignment between image regions and labels based on cosine similarity.
- Contrastive Learning: The embedding function is trained using a triplet loss. Crucially, the authors found that pulling negative samples from different images in the training batch significantly improved the robustness of the representations.
- Edge Attention Message Passing (EAMP): To leverage relational context, the authors introduce EAMP, a specialized Graph Neural Network that refines object features by incorporating information from neighboring nodes and edges.
2. Standard SGG Training
Once the matching is established, the resulting assignments serve as "pseudo-ground truth" labels. This allows the use of any off-the-shelf supervised SGG model (such as Neural Motifs or IMP) to be trained in a standard fashion, effectively bridging the gap between weak and full supervision.
Key Results
The proposed method was evaluated on the Visual Genome dataset (VG150 and VG200). The results demonstrate that this decoupled approach significantly moves the needle for the field.
As shown in the table above, WSGM achieves a substantial performance leap. In the Scene Graph Generation (SGGen) task, the instance-level recall ($R_{inst}$) jumped from 2.75% to 9.07%. Across various metrics and tasks (SGGen, SGCls, PredCls), the method consistently outperforms the previous state-of-the-art (VSPNet) by over 30%. Furthermore, when paired with the "Neural Motifs" architecture, the method achieved a Recall@100 of 6.45% in the weakly supervised setting.
Discussion and Conclusion
The success of this approach highlights that decoupling matching from generation is a more stable and effective strategy for weak supervision than joint optimization. By treating graph matching as a standalone contrastive learning problem, the model avoids the complexities of previous latent variable methods.
While a significant gap still remains between weakly-supervised and fully-supervised models (which achieve ~37% Recall@100), this work establishes a powerful new baseline. Future research may focus on improving object proposal quality specifically for weak supervision and exploring ways to further close the performance gap without increasing annotation costs.
Summary updated