Abstract
Vision-Language Pretraining (VLP) relies heavily on contrastive learning between image-text pairs. While "hard negatives" are essential for learning fine-grained alignments, they often introduce "false negatives"—unlabeled but semantically valid matches—that degrade model performance. We present FALCON, a learning-based strategy that dynamically selects negatives using a specialized scheduler. By balancing hardness and correctness, FALCON achieves state-of-the-art results across various benchmarks and architectures without relying on rigid heuristics.
Introduction: The Challenge of False Negatives
In contrastive VLP, models are trained to pull matching image-text pairs (positives) together in an embedding space while pushing non-matching pairs (negatives) apart. To learn effectively, models require "hard negatives": examples that are semantically similar to the anchor but technically incorrect. For instance, given the text "a woman holding a tennis racket," a hard negative might be an image of a man holding a tennis racket.
However, large-scale web datasets are inherently noisy. A "negative" image might actually be a valid description of the anchor text, creating a false negative. Aggressive mining of hard negatives often inadvertently selects these false negatives, providing contradictory signals that disrupt the learned embedding space. Existing solutions typically rely on static thresholds or expensive teacher models, which fail to adapt to the evolving needs of the model during training.
Methodology: Adaptive Negative Mining
FALCON (False-negative Aware Learning of COntrastive Negatives) replaces rigid heuristics with a Negative Mining Scheduler ($\pi_\phi$).
The Scheduler and Adaptive Mini-batching
The scheduler is a lightweight 4-layer MLP that analyzes the similarity distribution within a local search space. Instead of selecting specific items, it predicts an optimal "hardness quantile" ($q$) for each anchor. This allows for instance-level granularity: if an anchor is semantically simple, the scheduler may select high-$q$ (harder) negatives; if the anchor is complex or risky, it selects lower-$q$ (easier) negatives to avoid false matches.
Learning via Generative Feedback
The scheduler is trained using a novel proxy signal: the reduction in Masked Language Modeling (MLM) loss. If a specific set of negatives improves the model's ability to reconstruct text from visual context, those negatives are deemed informative, and the scheduler is rewarded. This bypasses the need for ground-truth labels for false negatives, allowing the system to learn in real-time from the model's own progress.
Experimental Results and Discussion
FALCON was evaluated on diverse datasets, including MSCOCO, Visual Genome, and the noisy DataComp-1M benchmark.
Performance and Robustness
FALCON consistently outperformed baseline methods like ALBEF and specialized strategies like MAFA. On MSCOCO Image-Text Retrieval, it achieved a 62.28 R@1, a significant improvement over the 55.60 baseline. Notably, on the noisy DataComp dataset, FALCON showed gains of 2.4% to 11.9%, demonstrating its ability to navigate real-world data noise.
Insights on Hardness
As shown in the training dynamics, a fixed high hardness quantile ($q=1.0$) leads to a rapid accumulation of false negatives, which can "break" the model's internal representations. Conversely, a very low $q$ provides insufficient challenge for the model to learn fine-grained features. FALCON's strength lies in its ability to dynamically adjust this quantile, treating hardness as a moving target that depends on both the specific instance and the current state of the model.
Conclusion
FALCON provides a robust, architecture-agnostic framework for improving Vision-Language Pretraining through intelligent negative mining. By leveraging generative loss as a teacher for a contrastive scheduler, it effectively mitigates the impact of false negatives in noisy datasets. Future work will explore scaling this approach to Large Vision-Language Models (LVLMs) and investigating alternative auxiliary tasks for the scheduler's reward signal.
Summary updated