TranstextNet: Transducing Text for Recognizing Unseen Visual Relationships

Gal Sadeh Kenigsfield, Ran El-YanivWACV 2021 · DOI

Introduction

Scene Graph Generation (SGG) is a fundamental task in computer vision that involves transforming an image into a structured representation where nodes represent objects and edges represent their interactions. While significant progress has been made in object detection, identifying the "verbs" or relationships between objects remains a major challenge. This difficulty stems primarily from data sparsity; standard training datasets are heavily biased toward common predicates (e.g., "on", "near"), leaving a "long tail" of rare but semantically rich relationships (e.g., "talking with", "holding").

An example of an image and its corresponding scene graph, illustrating complex interactions like "talking with" and "looking at".

The Recognition of Unseen Visual Relationships (ROUVR) task addresses this by requiring models to identify interactions that were never seen during visual training but may have been encountered in textual corpora. TranstextNet is proposed as a model-agnostic "plug-in" designed to bridge this gap by transducing knowledge from auxiliary text sources into the visual domain.

Methodology

TranstextNet introduces a novel Object-Relationship Mapping (ORM) layer that integrates textual knowledge into existing SGG architectures.

Textual Knowledge Acquisition

To overcome the limitations of visual training data, TranstextNet leverages massive auxiliary text sources, including image captions, ebooks from the Gutenberg Project, and synthetic text from GPT-2. By parsing these sources, the model builds a prior distribution of how objects typically interact in the real world, effectively gaining "common sense" that visual data alone cannot provide.

The ORM Plug-in and Neural Fusion

The ORM module can be integrated into standard SGG backbones. It processes the visual features of object pairs and fuses them with textual embeddings (e.g., GloVe) of potential relationships. Two primary fusion mechanisms are explored:

  1. Attention-based Fusion: Dynamically weights textual information based on the visual context of the object pair.
  2. Feature-wise Linear Modulation (FiLM): Uses textual information as a filter to scale and shift visual features, allowing the text to guide the visual interpretation.
The general architecture of a Scene Graph Generation backbone, which TranstextNet enhances with its ORM layer.

Zero-Shot Recognition via Cosine Loss

To enable the recognition of unseen relationships, the model employs a cosine loss function. This encourages the visual features to be projected into the same semantic space as word embeddings. Consequently, the model can predict a relationship label by finding the nearest word embedding, even if that specific relationship was never paired with those objects in the training images.

Experimental Setup

The model was evaluated on two benchmark datasets:

  • VG-200: A subset of Visual Genome with 150 object classes and 50 relationship classes.
  • VRD (Visual Relationship Detection): Featuring 5,000 images and 70 relationship classes, including 57 "unseen" relationships specifically for the ROUVR task.

Results and Discussion

TranstextNet achieved state-of-the-art performance across multiple metrics. On the VG-200 dataset, it significantly improved zero-shot SGG, raising Recall@50 (R@50) in the Predicate Classification (Pred-Cls) task from 10.9% to 26.7%. In the ROUVR task on the VRD dataset, the model successfully identified 40 out of 57 relationships that were entirely absent from the visual training set.

Furthermore, the model showed a marked improvement in mean Recall (mR@K), indicating its ability to handle the long-tail distribution of relationships more effectively than baseline models that tend to favor frequent, generic labels.

Conclusion

TranstextNet demonstrates that "reading" between the pixels—by leveraging the vast knowledge contained in text—is a viable strategy for improving visual relationship recognition. By aligning visual features with semantic word embeddings, the model gains the ability to generalize to unseen interactions. Future work will explore more sophisticated language models and the extension of this transduction approach to temporal relationship detection in video.

Summary updated

BibTeX
@inproceedings{kenigsfield2021transtextnet,
  author = {Gal Sadeh Kenigsfield and Ran El-Yaniv},
  title = {{TranstextNet: Transducing Text for Recognizing Unseen Visual Relationships}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00200},
}
Figure