Negative-Aware Attention Framework for Image-Text Matching

Kun Zhang, Zhendong Mao, Quan Wang, Yongdong Zhang

Introduction

Image-text matching is a fundamental task in cross-modal retrieval, aiming to measure the semantic similarity between visual and textual modalities. Traditional local-level matching methods typically employ attention mechanisms to align words with image regions. However, these models suffer from a significant "false-positive" problem: they are optimized to reward matches but fail to penalize mismatches.

Existing models often use functions like ReLU or LeakyReLU during the attention process, which effectively forces negative or low-relevance scores to zero. This creates a blind spot. For instance, if a caption mentions "football" but the image lacks it, standard models often ignore this discrepancy, leading to inflated similarity scores based solely on other matching elements like "boys" or "trees." The Negative-Aware Attention Framework (NAAF) addresses this by explicitly mining "mismatched fragments" to actively penalize similarity scores.

Conceptual comparison between existing methods and the proposed Negative-Aware Attention Framework (NAAF).

Methodology

The core innovation of NAAF is treating a mismatch not as noise to be filtered out, but as a high-value signal. The framework consists of two primary components designed to integrate these negative signals into the similarity inference.

Discriminative Mismatch Mining

Rather than relying on a fixed zero-threshold for relevance, NAAF adaptively learns an optimal boundary ($t_k$) to distinguish between matched and mismatched fragments. The framework models the similarity distributions of matched and mismatched fragments as two Gaussians and calculates a boundary that minimizes the probability of misclassification. This boundary is updated iteratively during training, allowing the model to become increasingly precise at identifying which words or regions truly do not belong together.

Neg-Pos Branch Matching

The framework splits the similarity calculation into two parallel paths:

  • Positive Branch: This follows the standard approach, focusing on aggregating matched fragments to compute a similarity score based on positive alignment.
  • Negative Branch: This branch specifically targets fragments (primarily words) that fall below the learned boundary $t_k$. These "mismatched clues" are used to calculate a dissimilarity degree, which then acts as a penalty to downgrade the overall score of the image-text pair.
The detailed architecture of NAAF, highlighting the Discriminative Mismatch Mining and the dual-branch matching process.

Experimental Evaluation

NAAF was evaluated on the Flickr30K and MSCOCO datasets, the two "gold standard" benchmarks for image-text matching. By focusing primarily on mismatched textual fragments—reasoning that captions are concise descriptions where every word matters—the model avoids interference from irrelevant background objects in images.

Key Results

The framework achieved state-of-the-art (SOTA) results across the board, notably outperforming previous mainstream models like SCAN and SGRAF:

  • Flickr30K: Achieved a Recall@1 (Image-to-Text) of 81.9%, a significant jump from the previous best of 77.8%.
  • MSCOCO (5K split): Achieved a Recall@1 (Image-to-Text) of 58.9%, setting a new benchmark for the field.

Qualitative analysis through visualization shows that NAAF correctly identifies "mismatch regions" (e.g., highlighting the area where a "reflective vest" or "football" should be but isn't), effectively "downgrading" false-positive candidates that fooled previous models.

Conclusion and Future Work

The Negative-Aware Attention Framework demonstrates that explicit mismatch mining is crucial for robust cross-modal retrieval. By adaptively learning boundaries and employing a dual-branch matching strategy, NAAF significantly reduces false positives. Future work will explore more sophisticated ways to distinguish between irrelevant background noise and meaningful image-side mismatches, as well as optimizing the framework for real-time search across massive datasets.

Summary updated

Abstract

Image-text matching, as a fundamental task, bridges the gap between vision and language. The key of this task is to accurately measure similarity between these two modalities. Prior work measuring this similarity mainly based on matched fragments (i.e., word/region with high relevance), while underestimating or even ignoring the effect of mismatched fragments (i.e., word/region with low relevance), e.g., via a typical LeaklyReLU or ReLU operation that forces negative scores close or exact to zero in attention. This work argues that mismatched textual fragments, which contain rich mismatching clues, are also crucial for image-text matching. We thereby propose a novel Negative-Aware Attention Framework (NAAF), which explicitly exploits both the positive effect of matched fragments and the negative effect of mismatched fragments to jointly infer image-text similarity. NAAF (1) delicately designs an iterative optimization method to maximally mine the mismatched fragments, facilitating more discriminative and robust negative effects, and (2) devises the two-branch matching mechanism to precisely calculate similarity/dissimilarity degrees for matched/mismatched fragments with different masks. Extensive experiments on two benchmark datasets, i.e., Flickr30K and MSCOCO, demonstrate the superior effectiveness of our NAAF, achieving state-of-the-art performance. Code will be released at: https://github.com/CrossmodalGroup/NAAF.
BibTeX
@inproceedings{zhang2022negative,
  author = {Kun Zhang and Zhendong Mao and Quan Wang and Yongdong Zhang},
  title = {{Negative-Aware Attention Framework for Image-Text Matching}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01521},
}
Figure