Introduction
While computer vision models have achieved significant success in single-image captioning—describing a scene like "a dog sitting on a bench"—real-world applications often demand a more complex understanding of how two images relate to one another. Whether it is a user requesting a photo edit ("remove the people from the background") or a security system identifying changes in surveillance footage, the ability to express visual relationships via language is a critical frontier in AI.
This paper introduces the Relational Speaker, a framework designed to move beyond standard captioning by explicitly modeling the transformation or logical connection between pairs of images. To support this research, the authors introduce a novel Image Editing Request dataset and a state-of-the-art Dynamic Relational Attention model.
The Challenge of Relational Reasoning
Standard captioning models are typically optimized for single visual inputs. When presented with two images, simple concatenation of features often fails because the model struggles to distinguish the "source" (before) from the "target" (after). Existing datasets for this task were previously limited in scope, often focusing on narrow domains like fixed-camera surveillance.
Methods: Dynamic Relational Attention
The core innovation of this work is the Dynamic Relational Attention mechanism. While previous attempts used static alignments—calculating relationships between image regions before generating text—the authors argue that this leads to information loss. Their proposed model calculates a "three-vector alignment" at every step of the language generation process:
- Source Image Features: Context from the original image.
- Target Image Features: Context from the modified or related image.
- Decoder State: The current state of the language model (LSTM).
By allowing the decoder to mediate the relationship between the two images dynamically, the model can focus on different types of changes—such as color shifts, object removals, or positional changes—at specific points in the generated sentence.
Diverse Benchmarking
The study evaluates the Relational Speaker across three distinct tasks to prove its general-purpose utility:
- Image Editing Request: A new dataset of 3,939 real-world image pairs from Reddit (r/photoshoprequest), featuring human-annotated instructions for edits.
- Spot-the-Diff: A dataset focused on identifying exhaustive differences in surveillance frames.
- NLVR2 (Natural Language Visual Reasoning): Originally a classification task, the authors converted this into a captioning challenge where the model must describe the logical relationship between two side-by-side images.
Key Results and Insights
The Dynamic Relational Attention model achieved state-of-the-art results across all benchmarks. On the Spot-the-Diff dataset, it reached a CIDEr score of 35.25, significantly outperforming the previous baseline of 32.80. On the new Image Editing Request dataset, METEOR scores improved from 11.58 to 12.80.
Beyond automated metrics, human evaluations confirmed the model's superiority. In head-to-head comparisons, human judges preferred captions from the full dynamic model over basic models by a wide margin (24% vs. 11% on the editing dataset), noting that the dynamic model was far more accurate in capturing specific changes.
Conclusion
The Relational Speaker demonstrates that capturing visual relationships requires more than just more data; it requires architectures that can dynamically align features across multiple inputs. While the model currently relies on high-level features from a ResNet-101 backbone, future work will explore pixel-level modeling to better handle highly specific or artistic editing requests that remain a challenge for current natural language generation.
Summary updated