Abstract
Image-text retrieval is a fundamental task to bridge the semantic gap between natural language and vision. Recent works primarily focus on aligning textual meanings with visual appearance. However, they often overlook the semantic discrepancy caused by syntactic structure in natural language expressions and relationships among visual entities. This oversight would lead to sub-optimal alignment and degraded retrieval performance, since the underlying semantic dependencies and object interactions remain inadequately encoded in both textual and visual embeddings. In this paper, we propose a novel Visual-Linguistic Dependency Encoding (VL-DE) framework, which explicitly models the dependency information among textual words and interaction patterns between image regions, improving the discriminative power of cross-modal representations for more accurate image-text retrieval. Specifically, VL-DE enhances textual representations by considering syntactic relationships and dependency types, and visual representations by attending to its spatially neighboring regions. Cross-attention mechanism is then introduced to aggregate aligned region-word pairs into image-text similarities. Analysis on Winoground, a dataset specially designed to measure vision-linguistic compositional structure reasoning, shows that VL-DE outperforms existing methods, demonstrating its effectiveness at this task. Comprehensive experiments on two benchmarks, Flickr30K and MS-COCO, further validates the competitiveness of our approach.
The Challenge of Compositional Reasoning
Most current image-text retrieval models excel at finding "bags of objects"—they can easily match a photo of a dog to a caption containing the word "dog." However, they often stumble on compositional reasoning.
Consider two sentences: "An adult kisses a child" and "A child kisses an adult." Both contain the exact same words, but the meaning is flipped. Standard models that treat words as independent fragments often produce nearly identical embeddings for these sentences, leading to "sub-optimal alignment." The same issue exists in vision: a "brown dog on a white couch" is visually distinct from a "white dog on a brown couch," even though the constituent entities are the same. This paper addresses the "semantic discrepancy" caused by ignoring syntactic structures in language and relational interactions in visual scenes.
The VL-DE Framework
The authors propose the VL-DE framework, which explicitly models the "who-does-what-to-whom" relationships by incorporating structural information into the embedding process.
The framework consists of three main modules:
- Textual Semantic Dependency Encoding: Instead of just using raw BERT embeddings, the model uses the Stanford CoreNLP toolkit to extract a dependency tree for each sentence. For every word, the model identifies its "parent" and "child" words and the specific type of relationship (e.g., nsubj for nominal subject, obj for object). These relationships are used to "enrich" the word's representation, ensuring that a word like "adult" carries the context of whether it is the subject or the object of the action.
- Visual Semantic Dependency Encoding: For images, the model uses a Faster R-CNN to detect salient regions. To capture context, it identifies the $K$ nearest spatial neighbors for each region. A gated mechanism then aggregates features from these neighbors, allowing the model to understand how an object interacts with its immediate surroundings (e.g., a "dog" feature is updated with information from the "couch" it is sitting on).
- Similarity Reasoning: To compare the enriched text and image features, the authors use a segment-wise cross-attention mechanism. They divide the high-dimensional feature vectors into smaller segments and calculate similarities across these segments, allowing the model to capture subtle, multi-perspective correspondences between modalities.
Experimental Evaluation
The model was evaluated on three key benchmarks:
- Winoground: A challenging "stress test" for compositional reasoning consisting of 800 image-caption pairs where captions use identical words in different orders.
- Flickr30K: A standard benchmark with 31,000 images and 155,000 sentences.
- MS-COCO: A large-scale benchmark with over 123,000 images, tested on both the 1K and 5K test splits.
Key Results
- Winoground Superiority: VL-DE achieved a 35.50% Text Score and 11.75% Group Score, significantly outperforming the previous state-of-the-art (CHAN at 29.75% and 8.75%, respectively). This demonstrates a much stronger grasp of linguistic composition.
- SOTA on Standard Benchmarks: On Flickr30K, the model achieved an R@1 of 83.7% for text retrieval, a notable improvement over existing ensemble models.
- Efficiency: Even without massive pre-training on billions of images (like CLIP), VL-DE's structural awareness allowed it to compete with and sometimes beat much larger pre-trained models on reasoning tasks.
Discussion and Conclusion
The core insight of this work is that syntax acts as a discriminator. Identical words should have different embeddings based on their syntactic role. By explicitly encoding the dependency type (e.g., amod for adjective modifier), the model learns that "brown" specifically modifies "dog," making the representation much more discriminative. Similarly, image regions require spatial context; a "ball" next to a "dog's mouth" is a specific semantic event that VL-DE's neighbor-aggregation captures.
While the model shows significant promise, it faces limitations. Its performance is tethered to the accuracy of external dependency parsing tools like Stanford CoreNLP. Furthermore, explicitly modeling neighbors and dependencies adds computational complexity compared to simpler global models. Future work will likely focus on improving the reliability of structured linguistic inputs and optimizing inference speed.
Overall, VL-DE demonstrates that local-level alignment—matching specific words to specific regions through the lens of their structural relationships—is superior to global-level alignment for complex retrieval tasks.
Summary updated