Contrastive learning trains an encoder by pulling two augmented views of the same image together and pushing every other image in the batch away. When the dataset contains many images of the same kind of thing, some of the pushed-away images, called "false negatives", should actually sit nearby. Current fixes make a yes/no decision for each negative. They cluster the data and drop same-cluster negatives, or they plug in a guess of how often two random images share a class. Our proposal is geometric and needs no such decision. Around each anchor, estimate the local "sheet" of nearby embeddings from a memory bank. Keep only the part of each negative's repulsion that points off that sheet. A same-class neighbour lies in the sheet and so receives almost no push. A different-class image is separated mainly across the sheet and keeps nearly its full push. No cluster count and no class prior are needed.
The proposal borrows from the generative-modelling literature on manifold overfitting. That literature shows that a density model trained on data lying on a thin lower-dimensional surface can drive its likelihood up without bound. It does this by squashing variance perpendicular to the surface, and in the process it learns where the data lives but not how probability mass is spread across it (Diagnosing and Fixing Manifold Overfitting in Deep Generative Models). We use this lens twice. First, it supplies the tangent/normal split that motivates the modified loss. Second, it gives a testable explanation for an open contradiction about whether detected false negatives should be removed or actively pulled together. Our hypothesis is that pulling them together is a contrastive analogue of manifold overfitting: it buys coarse-class accuracy by collapsing within-class variation.
Core Claim
We make two claims, one mechanistic and one diagnostic. The mechanistic claim concerns how false negatives do damage. They harm representations mainly through the component of the repulsive gradient that lies along the local data surface. Removing only that component, while keeping full repulsion in the normal directions, should match or beat discrete false-negative elimination. It should also preserve within-class structure better, without any clustering step.
The diagnostic claim explains the elimination-versus-attraction disagreement. Incremental False Negative Detection for Contrastive Learning (IFND) found that attraction ruins the embedding. Boosting Contrastive Self-Supervised Learning with False Negative Cancellation (FNC) found that attraction beats elimination, at roughly +1.75% versus +1.02% on ImageNet. We hypothesise that both observations are real and reflect a single trade-off. Attraction lowers the local intrinsic dimension of class regions. That helps a coarse linear probe when detection is precise. It is catastrophic when detection is noisy, and it always costs fine-grained, within-class information. Neither paper measured the geometry of its embeddings, so neither could see this trade-off.
Mechanism
Let $z_i$ be the unit-norm embedding of anchor $i$. Let $P_i$ be the orthogonal projector onto the top-$d_i$ principal directions of the anchor's $k$ nearest memory-bank embeddings, taken in the tangent space of the sphere at $z_i$. This is the "pancake" covariance of Manifold Parzen Windows, which estimated local tangent planes from $k$-nearest-neighbour SVD so that a density estimator would not leak mass off the data surface. We reuse their estimator in a different role: shaping a training gradient rather than defining a density.
Unlike that work, we do not fix a global dimension. Instead, $d_i$ is the smallest rank that explains a fixed fraction of local variance. This responds directly to the limitation that natural data has varying local dimension. The negative term of InfoNCE uses only the normal displacement:
$$ \mathcal{L}_i = -\log \frac{\exp(z_i^\top z_i^{+}/\tau)}{\exp(z_i^\top z_i^{+}/\tau) + \sum_{j} \exp\!\big(-\lVert (I - P_i)(z_i - z_j) \rVert^2 / 2\tau\big)} $$
Here $z_i^{+}$ is the positive view, $j$ ranges over negatives, $\tau$ is the temperature, and $P_i$ is held fixed (stop-gradient). A negative lying in the anchor's local sheet has zero normal displacement. Its gradient therefore vanishes, so it is neither repelled nor attracted, which is the "do not touch along the surface" behaviour. A negative off the sheet is repelled only perpendicular to it. The only difference from ordinary InfoNCE is that the in-sheet component of distance no longer contributes to repulsion.
Two contrasts with the closest prior work make the novelty concrete. The first is Contrastive Self-Supervised Learning As Neural Manifold Packing (CLAMP), which treats each image's augmented views as an ellipsoid and applies short-range repulsion whenever ellipsoids overlap. CLAMP shares our view that repulsion should be local and geometric. However, it repels any two distinct images that overlap, so it actively enforces false-negative interference. It also ignores ellipsoid orientation. We use the orientation of the semantic neighbourhood to decide which directions get repelled. The second contrast is Learning Manifold Data with Flow Matching. That paper proves the optimal generative velocity field splits into a tangent part and a linear normal contraction. It motivates treating the two subspaces differently, but it addresses sampling, not representation learning.
Research Program
The program has three parts.
Controlled diagnosis. We first settle the attraction-versus-elimination question under conditions no prior paper controlled. On ImageNet-100 and STL-10, we replace the learned false-negative detector with an oracle built from ground-truth labels. We then corrupt the oracle to precisions between 50% and 100% and train attraction (FNA-style) and elimination (FNE-style) at each level. Throughout training we track two geometric quantities on held-out embeddings. One is local intrinsic dimension (participation ratio of kNN covariance spectra). The other is within-class spread relative to between-class spread. We pair these with two kinds of evaluation. Coarse evaluation is a linear probe on the 100 classes. Fine evaluation is accuracy at distinguishing classes that share a WordNet parent, plus kNN retrieval within class.
The prediction is that attraction wins on the coarse probe only above some detection precision. It should lower local intrinsic dimension at every precision level, and it should lose on fine evaluation even at oracle precision. This would be a contrastive counterpart of Theorem 1 of the manifold overfitting paper: the right support is learned, but the distribution within it is lost.
Method. We train the tangent-aware loss above on a MoCo-style memory bank. The tangent estimate is turned on after a short warmup, because the noted sensitivity to early-stage representations applies to any neighbourhood estimator. We also report what happens with no warmup.
Second regime. We test the method on graphs. There, ProGCL: Rethinking Hard Negative Mining in Graph Contrastive Learning showed that message passing makes false negatives look like the hardest negatives, so the effect should be largest. We use Cora and ogbn-arXiv.
As a sanity substrate we use synthetic data: several curved class manifolds, such as swiss-roll and spiral sheets, embedded in high ambient dimension. This follows the latent-subspace setup of the flow-matching paper. Here we can measure directly whether within-class geodesic order survives training.
Baselines
The naive reference is plain InfoNCE (MoCo v2 / SimCLR style) with the same encoder, augmentations, epochs and memory bank, and a tuned temperature. Every other method must beat it.
The strongest discrete competitors are FNC and IFND. We run both inside our codebase, using the same backbone and epoch budget. Each gets the same number of hyperparameter trials as our method: FNC's support-set size and both its FNE and FNA variants, and IFND's cluster scales and acceptance schedule. For reference, FNC reports 74.4% ImageNet-1K linear top-1 and IFND reports 69.7%. Those numbers come from their own longer, multi-crop setups and are not directly comparable to our budget.
On ImageNet-100 the strongest reported result in the file is CLAMP's 85.12% linear top-1. We rerun CLAMP at matched views and epochs. Beating CLAMP means exceeding our rerun of it on the coarse probe while staying within noise of it on fine evaluation. Debiased contrastive learning (Debiased contrastive learning) is run with a $\tau$ sweep. It reports a 4.26% STL-10 gain over standard contrastive learning, which we treat as a relative target to reproduce before comparing.
On graphs, the bar is ProGCL-mix's reported 72.56% ogbn-arXiv test accuracy. Alongside it we run GDCL (Graph Debiased Contrastive Learning with Joint Representation Clustering), which reports 87.8% Cora linear node classification and 78.1% Cora clustering accuracy.
Three ablations isolate the key ingredient, which is directional rather than scalar handling of near neighbours.
- Soft distance down-weighting. We use the same $k$ neighbours but down-weight them isotropically, so repulsion keeps its full direction with reduced magnitude.
- Random projection. We replace $P_i$ with a random subspace of the same rank.
- Hard elimination. We remove the same $k$ neighbours from the denominator entirely.
If the first or third ablation matches the full method, the tangent geometry is not doing the work.
The method counts as a success if it does three things at matched compute. It must match the best discrete baseline on the coarse linear probe. It must beat all baselines on fine-grained and within-class retrieval metrics. And it must keep local intrinsic dimension closer to that of the plain InfoNCE embedding than attraction does.
Why not just…?
Why not just cluster with $k$-means and drop same-cluster negatives? This is essentially IFND. IFND needed multi-scale clustering with $k$ in {1000, 3000, 10000} and a hand-designed acceptance warmup before it beat PCL (69.7% vs 67.6%). That suggests a single partition captures semantic neighbourhoods poorly. Our method has no global $k$, and its notion of "neighbour" is local and directional. IFND is still a real competitor, and the head-to-head at matched budget decides it. If IFND ties us on both coarse and fine metrics, the geometric machinery is not justified.
Why not just use the debiased loss with a class prior? That loss applies one scalar correction everywhere. Its prior is ill-defined on long-tailed or continuous data. A uniform correction cannot distinguish a dense class region from a sparse one, whereas the pancake neighbourhood adapts to both, as Manifold Parzen Windows showed for density. The $\tau$ sweep at matched budget tests this. If a well-tuned prior matches us, the honest conclusion is that the local direction is unnecessary.
Why not just raise the temperature or soften near-neighbour weights? This is the most serious threat. Temperature already controls how strongly the closest negatives are repelled. Ablation 1 is exactly "soften without direction," and we sweep temperature for every method. If softening matches tangent projection, our directional story is wrong. The synthetic curved-manifold experiment should discriminate the two. Isotropic softening still pushes along the sheet and should break geodesic order. Projection should not.
Why not just use CLAMP, which is already geometric and short-range? CLAMP repels any overlap between distinct images, including same-class ones, and it ignores orientation by its authors' own account. It therefore should split same-class sheets just as InfoNCE does, only at shorter range. The fine-grained evaluation and the within-class geometry measurements test this directly.
Why not just add isotropic noise to embeddings, as done for flows on manifolds? Denoising Deep Generative Models found that noise-and-denoise only sporadically helps and is highly sensitive to noise scale. Spherical noise also leaks mass into empty directions, as Manifold Parzen Windows showed. We include a feature-jitter run at matched tuning budget, but we expect it to blur the line between hard and false negatives rather than help.
Decisive Checks
Three results would make or break the proposal.
The first is the oracle-precision sweep. If attraction lowers local intrinsic dimension and loses on fine evaluation even at 100% precision, while winning the coarse probe above some threshold, that reconciles IFND and FNC and confirms the manifold-overfitting reading. If attraction's geometry looks like elimination's, the reading is wrong and the diagnostic half of the proposal fails.
The second is the comparison against ablations 1 and 3 on ImageNet-100 and STL-10. Tangent projection must beat both on fine-grained metrics without losing the coarse probe. Otherwise direction does not matter.
The third is the graph regime. On Cora and ogbn-arXiv, where ProGCL showed hard negatives are disproportionately false, the gain over plain InfoNCE should be larger than on images. A smaller gain would indicate that our neighbourhood estimate is not tracking false negatives.
References
- Diagnosing and Fixing Manifold Overfitting in Deep Generative Models
- Incremental False Negative Detection for Contrastive Learning
- Boosting Contrastive Self-Supervised Learning with False Negative Cancellation
- Manifold Parzen Windows
- Contrastive Self-Supervised Learning As Neural Manifold Packing
- Learning Manifold Data with Flow Matching
- ProGCL: Rethinking Hard Negative Mining in Graph Contrastive Learning
- Debiased contrastive learning
- Graph Debiased Contrastive Learning with Joint Representation Clustering
- Denoising Deep Generative Models