Hardness as a Proposal, Not a Target: Importance-Corrected Hard Negatives for Contrastive RL

Contrastive reinforcement learning trains an agent without any hand-written reward. A classifier learns to tell which states actually occur later in the same trajectory (positives) apart from states drawn at random from other trajectories (negatives), and the classifier's score turns out to be a goal-conditioned value function. That second fact is what makes the approach useful. Its negatives, however, are uniform draws from the mini-batch, and most of them are far from the goal and easy to reject. Representation learning has a standard fix for easy negatives, which is to go looking for hard ones. In contrastive RL that fix is dangerous. The value-function interpretation holds only if the negatives follow the true distribution of states, and biasing toward hard negatives quietly changes what the critic learns.

This note proposes treating hardness as a way to sample negatives rather than a way to redefine the objective. Negatives are drawn from a large replay buffer with probability tilted toward states the critic currently confuses with the goal. Each negative is then reweighted by the ratio between the uniform and the tilted probability. The corrected loss has the same optimum as uniform contrastive RL, so the value interpretation survives, but it estimates the full-buffer contrastive loss with far lower variance than a handful of uniform negatives can. This has a consequence for hyperparameters. In existing hard-negative methods the hardness knob acts as a hidden objective weight that shifts the solution, which is why they are so sensitive to it. Under the correction it becomes a variance knob, and theory predicts both where its optimum sits and that performance should be flat around it.

Core Claim

Contrastive Learning as Goal-Conditioned Reinforcement Learning shows that an InfoNCE critic $f(s,a,g)=\phi(s,a)^\top\psi(g)$ recovers a goal-conditioned Q-function. InfoNCE is the softmax contrastive loss that scores one positive against a set of negatives. The recovery requires that the loss's denominator average over the marginal distribution $p(g)$ of goals. This paper supplies the reward-free setting that the rest of the proposal builds on.

In practice that marginal is approximated by the $B-1$ other goals in the batch. For a fixed state-action pair, the denominator is an estimate of the partition function

$$Z(s,a)=\mathbb{E}_{g^-\sim p}\left[\exp f(s,a,g^-)\right].$$

Suppose we draw $K$ negatives from a proposal $q(g^-\mid s,a)\propto p(g^-)\exp(\beta f(s,a,g^-))$ over a buffer of $M\gg B$ states, where $\beta\ge 0$ is the hardness tilt. We can then estimate $Z$ without bias as $\hat Z=\frac1K\sum_j \exp f(s,a,g_j^-)\,p(g_j^-)/q(g_j^-)$.

Standard importance-sampling theory says the variance-minimizing proposal for estimating $\mathbb{E}_p[\exp f]$ is $q\propto p\exp f$. The gradient of $\log Z$ weights samples by the same softmax, so this proposal is also a good one for the gradient. The first testable prediction follows: the best tilt should be near $\beta=1$. Values away from 1 should raise variance only, and should not move the learned values.

Uncorrected hard-negative sampling behaves differently. In the style of Contrastive Learning with Hard Negative Samples, which reweights negatives by $\exp(\beta f^\top f^-)$ with Positive-Unlabeled debiasing and treats the tilted distribution as the target, the critic's optimum is shifted by an amount that grows with $\beta$. The critic no longer encodes $\log p(g\mid s,a)/p(g)$.

Correction also handles false negatives, the recurring worry in the mining literature. In contrastive RL the marginal is supposed to contain states that are reachable from the anchor, because the critic estimates a density ratio and not a class boundary. A negative that happens to be a true future state is therefore not an error once it is weighted correctly. Uncorrected mining over-samples exactly those states and pushes true futures apart.

Research Program

We will build the estimator on top of the contrastive RL codebase. Each state in the replay buffer gets a cached goal embedding $\psi(g)$, refreshed on a rolling schedule in the manner of a MoCo queue.

At every training step, each anchor draws $K$ negatives from the tilted proposal over the cache. The cache can be searched by exact inner product at AntMaze scale, or by approximate nearest-neighbor search plus a uniform mixture component for larger buffers. The mixture keeps $q$ bounded away from zero, which keeps the importance weights bounded.

Staleness introduces no bias. The weights use the $q$ the negatives were actually drawn from, while $\exp f$ is computed with the current encoder. Staleness can only make the proposal worse, not the objective wrong. Self-normalized weights will be compared with the unbiased form, because self-normalization trades a small bias for lower variance and the right choice is an empirical question.

The analysis has three parts, and each one tests a stated prediction. First, we will directly measure bias and variance of the batch loss and its gradient against a reference computed over the entire buffer. This is expensive but feasible offline on AntMaze. Second, we will sweep $\beta$ and check whether corrected sampling gives a flat, U-shaped variance curve with its minimum near 1, while uncorrected sampling drifts in value accuracy as $\beta$ grows. Third, we will relate these measurements to control performance, asking whether lower-variance denominators sharpen value landscapes in long-horizon mazes where uniform negatives rarely land near the goal.

The multi-objective lens enters as a diagnosis and as a competitor, not as the method. Constrained Contrastive Reinforcement Learning adds a return-mined hard-negative loss to a temporal contrastive loss through a static weight, reports marked sensitivity to that weight, and hand-sets it to $\beta=1.5$. AMPO likewise reports sensitivity to its scale and temperature.

Our reading is that these methods mix a "true-marginal" objective with a "hard-marginal" objective and then have to arbitrate between the two. Gradient-balancing methods such as CAGrad, which finds an update maximizing worst-case improvement near the average gradient, and GAPO, which normalizes per-objective gradients, handle that arbitration after the fact. Importance correction removes the second objective altogether. The proposal therefore stands or falls on whether correction beats balancing.

Baselines

The do-nothing reference is contrastive RL exactly as published: uniform in-batch negatives at the paper's batch size. On offline AntMaze, Contrastive Learning as Goal-Conditioned Reinforcement Learning reports beating IQL and TD3+BC on 5 of 6 tasks, with a 7–9% absolute gain on the large mazes. The file gives no per-task normalized scores. We will therefore rerun it ourselves on the D4RL AntMaze datasets and report our own numbers alongside its published claims.

The strongest competitor for the same compute is uniform contrastive RL with a larger effective negative set, drawn uniformly from the same cached buffer with the same $K$ and without tilt. It isolates "more negatives from a cache" from "better-chosen negatives." It gets the same encoder, number of gradient steps, cache refresh schedule and per-method tuning budget, with 5 seeds each.

The second strong competitor is uncorrected tilted sampling, following Contrastive Learning with Hard Negative Samples, with its $\beta$ tuned over the same grid. It is the most direct transfer of representation-learning mining. That paper reports a 7.3% STL10 linear-probe gain over SimCLR and matching SimCLR's 400-epoch accuracy in 60 epochs, a reason to expect it to be competitive here.

We also include feature-mixed negatives from MoCHi, which reported 79.0% vs 78.0% ImageNet-100 top-1 over MoCo-v2. Synthetic mixed negatives have no density under $p$ and cannot be corrected, which makes MoCHi a natural foil for the value-equivalence argument.

A multi-objective baseline treats the uniform and the hard InfoNCE losses as two tasks and combines them with CAGrad, which on Meta-World MT50 reported 0.52 success vs 0.50 for PCGrad and 0.36 for multi-task SAC. We also run a static-weight sweep in the style of C2RL.

The key ablation is corrected versus uncorrected sampling at an identical tilt, proposal and $K$. Only the importance weights differ, so it isolates the proposal's single new ingredient.

Beating the baselines means two things. First, statistically higher normalized score or success on the large AntMaze mazes and on image-based Sawyer tasks than the compute-matched uniform-cache baseline. Second, a performance-versus-$\beta$ curve that stays within seed noise over a wide range, while the uncorrected and CAGrad variants degrade.

Why not just…?

Why not just use a bigger batch or a bigger uniform negative pool? This is the real threat. If the gains come only from more negatives, the uniform-cache baseline will match us, and we will report that outcome. The theory says the advantage of tilting is largest when $\exp f$ is concentrated on a small fraction of the buffer, as with long-horizon goals in large mazes. It should be small in short-horizon tasks. The bias/variance measurements against the full-buffer reference let us predict per task where each approach should win, before looking at control returns. Adversarial Contrastive Estimation argues that random negatives' denominator terms collapse toward zero, and MoCHi reports diminishing returns from larger memory banks. Both are suggestive but come from representation learning, so our experiment has to settle the question in control.

Why not just apply Robinson-style hard negative sampling unmodified? Its target is the tilted distribution, so in contrastive RL the critic stops estimating a density ratio. The damage should show up as a systematic, $\beta$-dependent error in learned values, which we measure directly on offline data where Monte Carlo reachability estimates are available. It should also show up as sensitivity to $\beta$, the same brittleness that paper itself acknowledges. If uncorrected sampling nonetheless wins on control, that would be an important negative result: value equivalence would matter less than Contrastive Learning as Goal-Conditioned Reinforcement Learning implies, consistent with its own finding that a filtering step needed for its proof hurt in practice.

Why not just treat hard and easy negatives as two losses and balance them with CAGrad or gradient normalization? Balancing chooses a point between two different fixed points. Correction has only one fixed point to choose. Balancing also adds a per-step dual solve, and RACO had to add weight clipping to stop over-correction toward minor objectives. The CAGrad baseline tests this directly. If it matches our method's returns and its insensitivity to hyperparameters, the simpler story is that balancing suffices. We would then reframe the contribution around the cheaper estimator.

Why not just use TD3 with hindsight relabeling? The contrastive RL paper reports that TD3+HER failed outright on image-based Sawyer Push and Bin where contrastive RL succeeded. We include it once for reference, but it is not the question here.

Proposed Experiments

Offline D4RL AntMaze, all six variants, is the primary testbed. There, the full-buffer reference loss is computable and the large mazes stress long-horizon value resolution.

Online image-based Sawyer Push and Bin test whether the gains survive a changing buffer and pixel encoders. The file gives no numeric success rates for these tasks. The single-goal exploration tasks from Demystifying The Mechanisms Behind Emergent Exploration in Goal-Conditioned RL test a side prediction. That paper attributes exploration to the denominator pushing visited-but-unrewarding states orthogonal to the goal. Tilted negatives should strengthen this effect, but the correction should keep it from overshooting. We measure it on its noisy-distractor task, where the paper reports irrelevant-area coverage of 0.36 for single-goal contrastive RL against 0.91 for PPO with random network distillation.

Three results decide the proposal, and all are fixed in advance. The first is the minimum of the estimator-variance curve near $\beta\approx1$. The second is corrected sampling beating the compute-matched uniform cache on the large mazes with flat $\beta$-sensitivity. The third is the corrected-vs-uncorrected ablation showing a measurable value bias only in the uncorrected arm. If the first holds but the second fails, the estimator is right but hardness does not matter for control, and we will say so.