Grounded Image Text Matching with Mismatched Relation Reasoning

Yu Wu, Yana Wei, Haozhe Wang, Yongfei Liu, Sibei Yang, Xuming He

Researchers from ShanghaiTech University and industry partners have introduced a new benchmark and a modular architecture designed to address a significant weakness in modern vision-language (VL) models: the inability to reason about relationships between objects, particularly in complex or low-data scenarios.

The GITM-MR Task

The authors propose Grounded Image Text Matching with Mismatched Relation (GITM-MR). This task goes beyond simple matching; it requires models to determine if a sentence matches an image, ground the mentioned objects, and, if a mismatch exists, pinpoint the exact relation that is incorrect.

The GITM-MR task requires identifying matches, grounding objects, or reasoning about mismatched relations.

Relation-sensitive Correspondence Reasoning Network (RCRN)

To tackle this challenge, the Relation-sensitive Correspondence Reasoning Network (RCRN) was developed. Unlike standard transformers that process text as a flat sequence, RCRN treats it as a structured program based on a Language Scene Graph (LSG). In this graph, nodes represent entities (e.g., "man", "bike") and edges represent relations (e.g., "wearing", "on").

Detailed overview of the RCRN architecture, showing how message propagation updates local beliefs.

Bi-directional Message Propagation

RCRN performs reasoning through two parallel passes in a "belief space" of visual-linguistic alignments:

  • Bottom-up (bp): Aggregates information from leaf nodes (attributes) up to the root (the main subject).
  • Top-down (td): Propagates context from the root back down to the leaves to resolve ambiguities (e.g., distinguishing between multiple similar objects).

This modularity allows the model to handle complex sentence structures by reusing primitive reasoning units (like similarity and aggregation modules) across different graph topologies.

Key Insights and Results

The Confidence Gap

A core discovery of this work is that a mismatched relation typically creates a "confidence gap" between its incident nodes in the graph. By identifying the edge with the largest inconsistency between match scores, the model can precisely locate the linguistic contradiction.

Performance and Generalization

RCRN significantly outperforms monolithic transformers like UNITER and FIBER in several key areas:

  • Length Generalization: When trained on short sentences and tested on long, complex ones (out-of-distribution), RCRN outperformed baselines by approximately 8-10% across matching, grounding, and reasoning tasks.
  • Data Efficiency: Due to its strong inductive bias from linguistic structure, RCRN can learn effectively from just 5-10% of the data required by standard large-scale models.
  • Interpretability: The graph-based approach makes the model's decision-making process traceable, allowing researchers to visualize which specific relations led to a match or mismatch prediction.

Conclusion

By moving away from "black-box" architectures toward structured, modular reasoning, RCRN provides a more robust and interpretable framework for vision-language understanding. While it currently relies on external pre-processors like object detectors, it sets a new standard for relational reasoning in the field.

Summary updated

BibTeX
@inproceedings{wu2023grounded,
  author = {Yu Wu and Yana Wei and Haozhe Wang and Yongfei Liu and Sibei Yang and Xuming He},
  title = {{Grounded Image Text Matching with Mismatched Relation Reasoning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00278},
}
Figure