Introduction
Scene Graph Generation (SGG) is a fundamental task in computer vision that involves transforming an image into a structured representation where nodes represent objects and edges represent their interactions. While significant progress has been made in object detection, identifying the "verbs" or relationships between objects remains a major challenge. This difficulty stems primarily from data sparsity; standard training datasets are heavily biased toward common predicates (e.g., "on", "near"), leaving a "long tail" of rare but semantically rich relationships (e.g., "talking with", "holding").
The Recognition of Unseen Visual Relationships (ROUVR) task addresses this by requiring models to identify interactions that were never seen during visual training but may have been encountered in textual corpora. TranstextNet is proposed as a model-agnostic "plug-in" designed to bridge this gap by transducing knowledge from auxiliary text sources into the visual domain.
Methodology
TranstextNet introduces a novel Object-Relationship Mapping (ORM) layer that integrates textual knowledge into existing SGG architectures.
Textual Knowledge Acquisition
To overcome the limitations of visual training data, TranstextNet leverages massive auxiliary text sources, including image captions, ebooks from the Gutenberg Project, and synthetic text from GPT-2. By parsing these sources, the model builds a prior distribution of how objects typically interact in the real world, effectively gaining "common sense" that visual data alone cannot provide.
The ORM Plug-in and Neural Fusion
The ORM module can be integrated into standard SGG backbones. It processes the visual features of object pairs and fuses them with textual embeddings (e.g., GloVe) of potential relationships. Two primary fusion mechanisms are explored:
- Attention-based Fusion: Dynamically weights textual information based on the visual context of the object pair.
- Feature-wise Linear Modulation (FiLM): Uses textual information as a filter to scale and shift visual features, allowing the text to guide the visual interpretation.
Zero-Shot Recognition via Cosine Loss
To enable the recognition of unseen relationships, the model employs a cosine loss function. This encourages the visual features to be projected into the same semantic space as word embeddings. Consequently, the model can predict a relationship label by finding the nearest word embedding, even if that specific relationship was never paired with those objects in the training images.
Experimental Setup
The model was evaluated on two benchmark datasets:
- VG-200: A subset of Visual Genome with 150 object classes and 50 relationship classes.
- VRD (Visual Relationship Detection): Featuring 5,000 images and 70 relationship classes, including 57 "unseen" relationships specifically for the ROUVR task.
Results and Discussion
TranstextNet achieved state-of-the-art performance across multiple metrics. On the VG-200 dataset, it significantly improved zero-shot SGG, raising Recall@50 (R@50) in the Predicate Classification (Pred-Cls) task from 10.9% to 26.7%. In the ROUVR task on the VRD dataset, the model successfully identified 40 out of 57 relationships that were entirely absent from the visual training set.
Furthermore, the model showed a marked improvement in mean Recall (mR@K), indicating its ability to handle the long-tail distribution of relationships more effectively than baseline models that tend to favor frequent, generic labels.
Conclusion
TranstextNet demonstrates that "reading" between the pixels—by leveraging the vast knowledge contained in text—is a viable strategy for improving visual relationship recognition. By aligning visual features with semantic word embeddings, the model gains the ability to generalize to unseen interactions. Future work will explore more sophisticated language models and the extension of this transduction approach to temporal relationship detection in video.
Summary updated