Abstract
Self-supervised contrastive learning (CL) has revolutionized representation learning but remains susceptible to data biases and the "false negative" problem. This paper introduces Weighted Contrastive Learning (WCL), a dual-encoder framework that intentionally trains one model to be biased. This biased model serves as a reference to help a primary, debiased model identify and prioritize learning from "hard" negatives—informative samples that are often ignored or misclassified due to dataset imbalances. WCL demonstrates significant improvements in accuracy, fairness, and robustness across image and tabular datasets.
Introduction
Contrastive learning operates on a simple premise: pull representations of the same image (positives) together and push representations of different images (negatives) apart. Despite its success, two major hurdles persist:
- The False Negative Problem: In the absence of labels, a "negative" sample may belong to the same semantic class as the anchor, leading the model to learn incorrect distinctions.
- The Hard Negative Problem: Randomly sampled negatives are often "easy" (e.g., distinguishing a car from a dog). Models gain the most information from "hard" negatives—samples that are visually similar but semantically distinct.
The authors posit that "hardness" is frequently a byproduct of data bias. For instance, if a dataset predominantly features birds against blue skies, a bird on water becomes a "bias-conflicting" sample. Standard CL models often treat these as hard negatives or ignore them entirely, resulting in poor generalization and unfairness toward minority groups.
Methodology: Embracing Bias to Debiasing
The core of the proposed method is the WCL architecture, which utilizes two identically structured encoders trained in parallel: a Biased Encoder ($E_b$) and a Debiased Encoder ($E_d$).
The Biased Encoder ($E_b$)
$E_b$ is trained using a novel self-supervised triplet loss. The authors theoretically prove that this loss function causes the model to focus on "easy" samples, effectively amplifying existing dataset biases. By becoming a "specialist" in majority patterns, $E_b$ provides a baseline for what a biased model finds "easy."
The Debiased Encoder ($E_d$)
$E_d$ is the primary model intended for downstream tasks. It uses $E_b$ as a compass. By comparing the Euclidean distances between an anchor and its negatives in both embedding spaces, the system calculates a relative difficulty weight ($w$). If $E_b$ perceives a negative sample as very close to the anchor (indicating it is either a hard negative or a bias-conflicting sample), $E_d$ assigns that sample a higher weight in its contrastive loss function ($L_{w-ctr}$). This mechanism forces $E_d$ to resolve the ambiguities that the biased model ignores.
Relative Distance Normalization
To ensure compatibility between the two encoders, the authors normalize distances within each batch. This allows for a robust measure of "relative difficulty" even if the two models operate in different embedding scales.
Experimental Results
WCL was evaluated across diverse benchmarks, including image classification (CIFAR-10/100), bias benchmarks (CelebA, Waterbirds), and fairness benchmarks (Adult, COMPAS).
Quantitative Performance
- Accuracy: On CIFAR-100, WCL achieved a top-1 accuracy of 69.09%, significantly higher than the standard SimCLR baseline (64.86%).
- Worst-Class Robustness: WCL showed dramatic improvements in "worst-class" accuracy. On CelebA, WCL reached 57.40% worst-group accuracy, a massive leap from the previous state-of-the-art HCL, which scored only 5.26%.
- Fairness: On the COMPAS recidivism dataset, WCL achieved the highest accuracy (64.3%) while maintaining a low Equalized Odds (EOd) violation (0.113), despite not having access to sensitive attributes during training.
Qualitative Analysis
t-SNE visualizations confirm that WCL produces a cleaner latent space compared to existing methods like DCL and HCL.
As shown in the visualizations, WCL results in tighter clusters with reduced intra-class variance, indicating that the model has learned more robust, debiased representations.
Conclusion
WCL offers a clever paradigm shift: instead of attempting to suppress bias directly, it uses a biased model as a high-contrast reference to guide a debiased model. While the dual-encoder approach doubles the computational requirements during pre-training and introduces sensitivity to the class probability hyperparameter ($\tau^+$), its gains in robustness and fairness are substantial. Future work will focus on scaling this approach to massive datasets like ImageNet-1k.
Summary updated