Abstract
Most existing image-text matching methods adopt triplet loss as the optimization objective. Choosing proper negative samples—specifically hard negatives—is crucial for effective model training. However, existing methods often treat the most similar samples as hard negatives, even though they may be "false negatives" that share positive semantic associations with the anchor. Repelling these false negatives misleads semantic representation learning. This paper proposes a False Negative Elimination (FNE) strategy to select negatives via Bayesian sampling. By constructing similarity distributions for positive and negative samples, the model calculates the probability that a sample is a false negative and adjusts its sampling weight accordingly. To handle small batch sizes, a momentum memory module retains a large negative buffer. Experiments on Flickr30K and MS-COCO demonstrate that FNE significantly improves retrieval performance over state-of-the-art methods.
Introduction
Image-text matching (ITM) is a cornerstone of modern multi-modal AI, enabling applications from cross-modal search to automated captioning. The primary goal is to learn a shared embedding space where relevant image-text pairs are positioned closely together. Traditionally, this is achieved using triplet loss, which optimizes the distance between an anchor, a positive sample, and a negative sample.
To accelerate convergence and improve discriminative power, researchers typically employ "hard negative mining," focusing on the negative samples most similar to the anchor. However, this paper identifies a fundamental flaw in this approach: in large-scale datasets, the "hardest" negatives are frequently semantically consistent with the anchor, making them false negatives. Forcing the model to repel these samples results in "blurry" representations and degraded retrieval accuracy.
The False Negative Elimination (FNE) Framework
The authors propose the False Negative Elimination (FNE) strategy to mitigate the impact of these misleading samples. The framework consists of three core components designed to refine the training process without requiring manual labeling of false negatives.
1. Bayesian False Negative Estimation
Rather than using a rigid similarity threshold to identify false negatives, FNE employs a probabilistic approach based on Bayes' rule. The model tracks the distributions of similarity scores for known positive and negative pairs during training. When evaluating a potential negative sample, the model calculates a posterior probability: the likelihood that the sample is actually a positive match given its similarity to the anchor. This allows the system to dynamically adapt as the model’s feature extraction capabilities improve.
2. Weighted Sampling and the "Cut-down" Strategy
Once the false negative probability is established, it is used to weight the sampling process:
- False Negative Suppression: Samples with a high probability of being false negatives are assigned lower weights, preventing them from disrupting the learning of correct semantic associations.
- Easy Negative Cut-down: To ensure the model remains focused on informative samples, "easy" negatives (those with very low similarity to the anchor) are also down-weighted. This "cut-down" strategy forces the model to prioritize "true hard negatives"—samples that are challenging but semantically distinct from the anchor.
3. Momentum Memory Module
High-performance ITM models, such as those utilizing Vision Transformers (ViT) or BERT, are computationally expensive, often limiting training to small batch sizes. Small batches rarely contain enough hard negatives or false negatives to effectively train the FNE module. To overcome this, the authors implement a large, queue-based memory buffer (8,192 samples). A momentum encoder ensures that the features stored in this buffer remain consistent with the evolving main model, providing a rich and diverse pool for negative sampling.
Experimental Evaluation
The FNE strategy was rigorously tested on the Flickr30K and MS-COCO benchmarks. The results indicate that FNE consistently outperforms previous state-of-the-art (SOTA) methods like NAAF and RVSE++.
- Flickr30K: The model achieved a Recall@1 (R@1) of 85.4% for image-to-text retrieval, a significant improvement over the previous best of 83.6%.
- MS-COCO (5K Split): On the more challenging 5K test split, FNE reached 64.3% R@1 for image-to-text retrieval, outperforming existing SOTA models by nearly 3%.
Beyond raw metrics, visualization of cross-attention maps revealed that models trained with FNE maintain a much sharper focus on relevant objects (e.g., specific animals or vehicles) compared to models forced to repel false negatives, which often exhibit scattered and confused attention patterns.
Conclusion and Future Work
The FNE framework demonstrates that in the context of image-text matching, the "hardest" negative is not always the best teacher. By identifying and eliminating the influence of false negatives through Bayesian sampling and a momentum memory buffer, FNE achieves superior retrieval performance while maintaining the efficiency of Visual Semantic Embedding (VSE) architectures.
While the method introduces some hyperparameter sensitivity regarding sampling density and requires a small amount of additional VRAM for the memory buffer, its success suggests broad applicability. Future research may extend this Bayesian elimination strategy to other contrastive learning domains, such as self-supervised image representation learning, where the false negative problem remains a persistent challenge.
Summary updated