Your Negative May not Be True Negative: Boosting Image-Text Matching with False Negative Elimination

Haoxuan Li, Yi Bin, Junrong Liao, Yang Yang, Heng Tao Shen

Abstract

Most existing image-text matching methods adopt triplet loss as the optimization objective. Choosing proper negative samples—specifically hard negatives—is crucial for effective model training. However, existing methods often treat the most similar samples as hard negatives, even though they may be "false negatives" that share positive semantic associations with the anchor. Repelling these false negatives misleads semantic representation learning. This paper proposes a False Negative Elimination (FNE) strategy to select negatives via Bayesian sampling. By constructing similarity distributions for positive and negative samples, the model calculates the probability that a sample is a false negative and adjusts its sampling weight accordingly. To handle small batch sizes, a momentum memory module retains a large negative buffer. Experiments on Flickr30K and MS-COCO demonstrate that FNE significantly improves retrieval performance over state-of-the-art methods.

Introduction

Image-text matching (ITM) is a cornerstone of modern multi-modal AI, enabling applications from cross-modal search to automated captioning. The primary goal is to learn a shared embedding space where relevant image-text pairs are positioned closely together. Traditionally, this is achieved using triplet loss, which optimizes the distance between an anchor, a positive sample, and a negative sample.

To accelerate convergence and improve discriminative power, researchers typically employ "hard negative mining," focusing on the negative samples most similar to the anchor. However, this paper identifies a fundamental flaw in this approach: in large-scale datasets, the "hardest" negatives are frequently semantically consistent with the anchor, making them false negatives. Forcing the model to repel these samples results in "blurry" representations and degraded retrieval accuracy.

The False Negative Elimination (FNE) Framework

The authors propose the False Negative Elimination (FNE) strategy to mitigate the impact of these misleading samples. The framework consists of three core components designed to refine the training process without requiring manual labeling of false negatives.

1. Bayesian False Negative Estimation

Rather than using a rigid similarity threshold to identify false negatives, FNE employs a probabilistic approach based on Bayes' rule. The model tracks the distributions of similarity scores for known positive and negative pairs during training. When evaluating a potential negative sample, the model calculates a posterior probability: the likelihood that the sample is actually a positive match given its similarity to the anchor. This allows the system to dynamically adapt as the model’s feature extraction capabilities improve.

2. Weighted Sampling and the "Cut-down" Strategy

Once the false negative probability is established, it is used to weight the sampling process:

  • False Negative Suppression: Samples with a high probability of being false negatives are assigned lower weights, preventing them from disrupting the learning of correct semantic associations.
  • Easy Negative Cut-down: To ensure the model remains focused on informative samples, "easy" negatives (those with very low similarity to the anchor) are also down-weighted. This "cut-down" strategy forces the model to prioritize "true hard negatives"—samples that are challenging but semantically distinct from the anchor.

3. Momentum Memory Module

High-performance ITM models, such as those utilizing Vision Transformers (ViT) or BERT, are computationally expensive, often limiting training to small batch sizes. Small batches rarely contain enough hard negatives or false negatives to effectively train the FNE module. To overcome this, the authors implement a large, queue-based memory buffer (8,192 samples). A momentum encoder ensures that the features stored in this buffer remain consistent with the evolving main model, providing a rich and diverse pool for negative sampling.

Experimental Evaluation

The FNE strategy was rigorously tested on the Flickr30K and MS-COCO benchmarks. The results indicate that FNE consistently outperforms previous state-of-the-art (SOTA) methods like NAAF and RVSE++.

  • Flickr30K: The model achieved a Recall@1 (R@1) of 85.4% for image-to-text retrieval, a significant improvement over the previous best of 83.6%.
  • MS-COCO (5K Split): On the more challenging 5K test split, FNE reached 64.3% R@1 for image-to-text retrieval, outperforming existing SOTA models by nearly 3%.

Beyond raw metrics, visualization of cross-attention maps revealed that models trained with FNE maintain a much sharper focus on relevant objects (e.g., specific animals or vehicles) compared to models forced to repel false negatives, which often exhibit scattered and confused attention patterns.

Conclusion and Future Work

The FNE framework demonstrates that in the context of image-text matching, the "hardest" negative is not always the best teacher. By identifying and eliminating the influence of false negatives through Bayesian sampling and a momentum memory buffer, FNE achieves superior retrieval performance while maintaining the efficiency of Visual Semantic Embedding (VSE) architectures.

While the method introduces some hyperparameter sensitivity regarding sampling density and requires a small amount of additional VRAM for the memory buffer, its success suggests broad applicability. Future research may extend this Bayesian elimination strategy to other contrastive learning domains, such as self-supervised image representation learning, where the false negative problem remains a persistent challenge.

Summary updated

Abstract

Most existing image-text matching methods adopt triplet loss as the optimization objective, and choosing a proper negative sample for the triplet of <anchor, positive, negative> is important for effectively training the model, e.g., hard negatives make the model learn efficiently and effectively. However, we observe that existing methods mainly employ the most similar samples as hard negatives, which may not be true negatives. In other words, the samples with high similarity but not paired with the anchor may reserve positive semantic associations, and we call them false negatives. Repelling these false negatives in triplet loss would mislead the semantic representation learning and result in inferior retrieval performance. In this paper, we propose a novel False Negative Elimination (FNE) strategy to select negatives via sampling, which could alleviate the problem introduced by false negatives. Specifically, we first construct the distributions of positive and negative samples separately via their similarities with the anchor, based on the features extracted from image and text encoders. Then we calculate the false negative probability of a given sample based on its similarity with the anchor and the above distributions via the Bayes' rule, which is employed as the sampling weight during negative sampling process. Since there may not exist any false negative in a small batch size, we design a memory module with momentum to retain a large negative buffer and implement our negative sampling strategy spanning over the buffer. In addition, to make the model focus on hard negatives, we reassign the sampling weights for the simple negatives with a cut-down strategy. The extensive experiments are conducted on Flickr30K and MS-COCO, and the results demonstrate the superiority of our proposed false negative elimination strategy. The code is available at https://github.com/LuminosityX/FNE.
BibTeX
@inproceedings{li2023your,
  author = {Haoxuan Li and Yi Bin and Junrong Liao and Yang Yang and Heng Tao Shen},
  title = {{Your Negative May not Be True Negative: Boosting Image-Text Matching with False Negative Elimination}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612101},
}
Figure