Abstract
Contrastive learning has become a key component of self-supervised learning approaches for computer vision. By learning to embed two augmented versions of the same image close to each other and to push the embeddings of different images apart, one can train highly transferable visual representations. As revealed by recent studies, heavy data augmentation and large sets of negatives are both crucial in learning such representations. At the same time, data mixing strategies either at the image or the feature level improve both supervised and semi-supervised learning by synthesizing novel examples, forcing networks to learn more robust features. In this paper, we argue that an important aspect of contrastive learning, i.e., the effect of hard negatives, has so far been neglected. To get more meaningful negative samples, current top contrastive self-supervised learning approaches either substantially increase the batch sizes, or keep very large memory banks; increasing the memory size, however, leads to diminishing returns in terms of performance. We therefore start by delving deeper into a top-performing framework and show evidence that harder negatives are needed to facilitate better and faster learning. Based on these observations, and motivated by the success of data mixing, we propose hard negative mixing strategies at the feature level, that can be computed on-the-fly with a minimal computational overhead. We exhaustively ablate our approach on linear classification, object detection and instance segmentation and show that employing our hard negative mixing procedure improves the quality of visual representations learned by a state-of-the-art self-supervised learning method.
The Challenge of Hard Negatives
In contrastive learning frameworks like MoCo or SimCLR, models learn by pulling "positive" pairs (different views of the same image) together and pushing "negative" pairs (views of different images) apart in an embedding space. The effectiveness of this learning process depends heavily on the "hardness" of the negatives. If a negative is too easy—for instance, a model easily distinguishing a dog from a car—the gradient signal is weak, and the model learns little.
Currently, state-of-the-art methods rely on massive batch sizes or large memory banks to find these elusive hard negatives. However, as the model improves, most negatives in a static bank become "easy," leading to diminishing returns. This paper proposes a shift in strategy: instead of merely searching for hard negatives, we can synthetically manufacture them.
MoCHi: Mixing of Contrastive Hard Negatives
The authors introduce MoCHi, a strategy that operates at the feature level (embedding space) rather than the pixel level. Building on the MoCo-v2 framework, MoCHi introduces two primary mixing strategies to create challenging synthetic negatives on-the-fly:
- Mixing Hard Negatives: The algorithm identifies the $N$ "hardest" negatives currently in the memory bank (those with the highest cosine similarity to the query). It then creates $s$ new synthetic negatives by taking random convex combinations of pairs from these $N$ samples.
- Mixing with the Query: To create even more difficult samples, MoCHi generates $s'$ additional negatives by mixing the hardest negatives with the query itself. Since these synthetic points contain features from the query, they are exceptionally hard for the model to distinguish from the true positive pair.
Key Insights and Results
Proxy Task vs. Transfer Performance
A core finding of this work is that making the "proxy task" (the contrastive game) harder for the model actually improves the quality of the learned features for downstream tasks. As shown in the training curves, MoCHi models often exhibit lower accuracy on the training proxy task compared to standard MoCo-v2, yet they achieve superior performance in real-world applications like object detection.
Efficiency and Uniformity
MoCHi significantly improves the uniformity of the embedding distribution on the hypersphere. By populating the space around each query with synthetic negatives, the model is forced to distribute representations more evenly, which correlates with better generalization.
Empirically, this leads to massive gains in training efficiency:
- Faster Learning: On PASCAL VOC, MoCHi pre-trained for only 200 epochs achieved results comparable to a MoCo-v2 baseline pre-trained for 800 epochs—a 4x improvement in efficiency.
- State-of-the-Art Transfer: On the COCO dataset, MoCHi consistently outperformed MoCo-v2. For instance segmentation, MoCHi pre-trained for 100 epochs matched the performance of a fully supervised model.
- ImageNet-100: MoCHi improved linear classification accuracy by ~1% over the strong MoCo-v2 baseline.
Limitations and Future Work
While MoCHi shows significant gains in transfer learning for detection and segmentation, the improvements in linear classification on the full ImageNet-1K were less pronounced, possibly due to task-specific biases. The method also introduces new hyperparameters ($N, s, s'$), suggesting that a "curriculum learning" approach—adjusting the hardness of synthetic negatives as training progresses—could be a fruitful path for future research to further stabilize and accelerate learning.
Summary updated