Expressing Visual Relationships via Language

Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, Mohit Bansal

Introduction

While computer vision models have achieved significant success in single-image captioning—describing a scene like "a dog sitting on a bench"—real-world applications often demand a more complex understanding of how two images relate to one another. Whether it is a user requesting a photo edit ("remove the people from the background") or a security system identifying changes in surveillance footage, the ability to express visual relationships via language is a critical frontier in AI.

This paper introduces the Relational Speaker, a framework designed to move beyond standard captioning by explicitly modeling the transformation or logical connection between pairs of images. To support this research, the authors introduce a novel Image Editing Request dataset and a state-of-the-art Dynamic Relational Attention model.

The Relational Speaker framework generating an editing instruction from a pair of images.

The Challenge of Relational Reasoning

Standard captioning models are typically optimized for single visual inputs. When presented with two images, simple concatenation of features often fails because the model struggles to distinguish the "source" (before) from the "target" (after). Existing datasets for this task were previously limited in scope, often focusing on narrow domains like fixed-camera surveillance.

Methods: Dynamic Relational Attention

The core innovation of this work is the Dynamic Relational Attention mechanism. While previous attempts used static alignments—calculating relationships between image regions before generating text—the authors argue that this leads to information loss. Their proposed model calculates a "three-vector alignment" at every step of the language generation process:

  1. Source Image Features: Context from the original image.
  2. Target Image Features: Context from the modified or related image.
  3. Decoder State: The current state of the language model (LSTM).

By allowing the decoder to mediate the relationship between the two images dynamically, the model can focus on different types of changes—such as color shifts, object removals, or positional changes—at specific points in the generated sentence.

Diverse Benchmarking

The study evaluates the Relational Speaker across three distinct tasks to prove its general-purpose utility:

  • Image Editing Request: A new dataset of 3,939 real-world image pairs from Reddit (r/photoshoprequest), featuring human-annotated instructions for edits.
  • Spot-the-Diff: A dataset focused on identifying exhaustive differences in surveillance frames.
  • NLVR2 (Natural Language Visual Reasoning): Originally a classification task, the authors converted this into a captioning challenge where the model must describe the logical relationship between two side-by-side images.
Examples from the NLVR2 dataset showing complex logical relationships between image pairs.

Key Results and Insights

The Dynamic Relational Attention model achieved state-of-the-art results across all benchmarks. On the Spot-the-Diff dataset, it reached a CIDEr score of 35.25, significantly outperforming the previous baseline of 32.80. On the new Image Editing Request dataset, METEOR scores improved from 11.58 to 12.80.

Beyond automated metrics, human evaluations confirmed the model's superiority. In head-to-head comparisons, human judges preferred captions from the full dynamic model over basic models by a wide margin (24% vs. 11% on the editing dataset), noting that the dynamic model was far more accurate in capturing specific changes.

Conclusion

The Relational Speaker demonstrates that capturing visual relationships requires more than just more data; it requires architectures that can dynamically align features across multiple inputs. While the model currently relies on high-level features from a ResNet-101 backbone, future work will explore pixel-level modeling to better handle highly specific or artistic editing requests that remain a challenge for current natural language generation.

Summary updated

Abstract

Describing images with text is a fundamental problem in vision-language research. Current studies in this domain mostly focus on single image captioning. However, in various real applications (e.g., image editing, difference interpretation, and retrieval), generating relational captions for two images, can also be very useful. This important problem has not been explored mostly due to lack of datasets and effective models. To push forward the research in this direction, we first introduce a new language-guided image editing dataset that contains a large number of real image pairs with corresponding editing instructions. We then propose a new relational speaker model based on an encoder-decoder architecture with static relational attention and sequential multi-head attention. We also extend the model with dynamic relational attention, which calculates visual alignment while decoding. Our models are evaluated on our newly collected and two public datasets consisting of image pairs annotated with relationship sentences. Experimental results, based on both automatic and human evaluation, demonstrate that our model outperforms all baselines and existing methods on all the datasets.
BibTeX
@inproceedings{tan2019expressing,
  author = {Hao Tan and Franck Dernoncourt and Zhe Lin and Trung Bui and Mohit Bansal},
  title = {{Expressing Visual Relationships via Language}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1182},
}
Figure