An agent that must adapt to a new task from a handful of experience samples usually relies on a task encoder: a network that reads a short context of transitions (state, action, reward, next state) and outputs a code $z$ describing which task it is facing. The known failure is that the encoder also absorbs how the data was collected, meaning the quirks of the policy that generated the context. At test time the agent's own behavior differs from that policy, the code drifts, and performance collapses. On real robots a second nuisance joins the first. Sensor noise depends on how the robot moves, since fast motion blurs cameras and smears range readings. Behavior and noise therefore become entangled, and an encoder that learns to ignore one can end up keying on the other.
We propose to build the "wrong answers" for contrastive training by retrieval instead of generation. For each training transition, we search a large offline bank of other tasks for transitions that start from nearly the same state and action, and we use their outcomes as negatives. Because each negative shares the anchor's state and action, the encoder can tell the anchor apart from them only by reading the outcome, which carries the task, and never by reading behavior. Retrieval also reverses the usual scaling story. Methods that generate negatives with a model get more expensive as the task space grows. Retrieval gets more informative, because a denser bank of tasks contains more near-identical starting points with different outcomes. We borrow the indexing and grouping tools of extreme multi-label classification, the field that handles output spaces of millions of labels, so that each anchor touches only $O(k)$ candidate tasks rather than all of them.
Core Claim
Our central claim is that state-action-matched negatives retrieved from a large task bank yield task codes that stay invariant both to the behavior policy and to behavior-correlated sensor noise. Their quality should rise with the number of training tasks, while the cost per update stays independent of it. The closest work is CORRO (Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning). CORRO showed that splitting contexts into single transitions and contrasting against negatives that share $(s,a)$ removes behavioral signatures. Its negatives, however, are hallucinated by a conditional VAE or produced by randomizing rewards, and the file notes that this struggles in high dimensions and adds substantial cost. CORRO also assumes clean state vectors and only a handful of tasks. We keep CORRO's principle of matched negatives and change how negatives are produced, how far the method scales, and the noise conditions under which it is trained and tested.
Concretely, take an anchor transition $x=(\tilde s, a, r, \tilde s')$ from task $i$, where $\tilde s$ is the noisy observation. We retrieve, from tasks $j \neq i$, transitions whose filtered state-action key lies near $(\tilde s, a)$. We then form a spliced negative $x_j^- = (\tilde s, a, r_j, \tilde s + \Delta_j)$, where $r_j$ is the reward and $\Delta_j$ the state change observed in the retrieved transition. The positive $x^+$ comes from the same task $i$, but is collected by a different behavior policy under an independent noise realization. With $f$ the transition encoder, $\tau$ a temperature and $\mathcal N(x)$ the retrieved neighbor set of size $k$, the loss is
$$ \mathcal L(x) = -\log \frac{\exp(f(x)^\top f(x^+)/\tau)}{\exp(f(x)^\top f(x^+)/\tau) + \sum_{j \in \mathcal N(x)} \exp(f(x)^\top f(x_j^-)/\tau)}. $$
The positive pairs differ in both behavior and noise, while the negatives agree with the anchor on everything except the outcome. A code that minimizes this loss therefore has no use for behavior or noise. Splicing the state change $\Delta_j$ is a first-order approximation that is exact when nearby states respond alike. Measuring how well that approximation holds is part of the work.
Research Program
The work has three parts. The first is the retrieval index. Task banks are grown from tens to tens of thousands of tasks by sampling parametric goals and dynamics densely. Transitions are indexed by state-action key, and tasks are partitioned into groups so that each anchor draws its negatives from a few groups of tasks with similar outcomes plus a small uniform sample. This mirrors the head-tail split and group-shared structure of HASTE, which reached 8.6M labels on 12.5 GiB versus 105.6 GiB dense. It also mirrors the tree partitioning in Probabilistic Label Trees. We reuse their idea of structured candidate sets, not their kernels. The Linear Scaling Bottleneck in Large Label Spaces motivates the constraint that no step may normalize over all tasks.
The second part is the noise model. Clean MuJoCo states are corrupted with three kinds of noise: independent Gaussian noise, slow calibration drift, and velocity-proportional noise that imitates motion blur. The third kind is the critical one, because it makes the noise itself carry behavior information. We expect information-minimization methods such as CSRO to fail here. CSRO's upper-bound penalty on $I(z;(s,a))$ is computed on noisy $\tilde s$, so it can treat noise variance as behavior, or it can let drift leak in as a shortcut that predicts the task.
The third part is analysis. We train linear probes that try to predict, from $z$, the identity of the behavior policy and the noise level. We also measure splice error directly, by comparing spliced outcomes with the true outcomes in simulation.
Baselines
The naive reference point is Offline PEARL and FOCAL, which apply no invariance mechanism. In CORRO's paper their returns fall to roughly -200 on Half-Cheetah-Vel under out-of-distribution (OOD) exploration. The most direct competitor is CORRO itself, which reports -89.7 on the same setting. It is the strongest test of our specific contribution because it shares our matched-negative principle and differs only in how negatives are produced. We will rerun CORRO with both its CVAE negatives and its reward-randomization negatives.
CSRO represents information minimization. It reports -6.4 on Point-Robot against -11.8 for FOCAL. MetaSTAR (Behavior-Invariant Task Representation Learning with Transformer-based World Models for Offline Meta-RL) is the strongest reported method overall. It reaches 296 on Cheetah-Vel-Sparse versus 216 for CORRO and holds 59 on Ant-Dir OOD, at roughly 15 versus 8 hours of training. We compare against it on both return and wall-clock time.
All methods share the same offline datasets, behavior-policy splits, noise processes, downstream offline RL learner and hyperparameter search budget. We report GPU-hours for each. Beating CORRO means higher OOD-behavior return at equal or lower compute, both in the clean setting and under velocity-proportional noise. Against MetaSTAR, matching its return at model-free cost would count as success.
Two ablations isolate the key ingredient. The first replaces retrieved matched negatives with random in-batch negatives drawn from other tasks. The second keeps retrieval but removes cross-noise positives. On the large-task-count and noisy settings no reported result was found on Lacuna for any baseline. Those numbers will all be new.
Why not just…?
Why not just use in-batch negatives from other tasks, which already scale? This is the most serious threat, because sampled contrastive losses avoid the linear cost entirely. The weakness is that unmatched negatives differ from the anchor in $(s,a)$ as well as in outcome. The encoder can then separate tasks by their state-action distribution, which is exactly the behavioral shortcut that CORRO's matched design was built to remove. At ten tasks in-batch negatives may well suffice. Our task-count sweep will settle the question. If the random-negative ablation matches us at 10,000 tasks under OOD behavior and its behavior probe stays at chance, the retrieval mechanism is unnecessary and we will report that.
Why not just add noise augmentation or domain randomization to CORRO or CSRO? DROPO found that uniform randomization left about 10 cm error on Franka pushing, against about 2 cm for calibrated distributions. Augmentation also does not stop noise that correlates with velocity from encoding behavior. We include CORRO+augmentation and CSRO+augmentation as baselines, so this alternative is tested directly.
Why not just classify task labels with a supervised head, using sampled softmax at scale? CORRO reports that it beats a ground-truth-label supervised encoder on Ant-Dir. Labels also say nothing about which features are task-relevant rather than behavioral. We run supervised classification with sampled softmax as a baseline and probe its codes for behavior leakage.
Why not just denoise with a VAE first and then run an existing meta-RL method? The embedded autoencoder work shows that such latents speed single-task learning 4–5×. Those latents are blind to both task and behavior, so the behavior leakage passes through untouched. We test VAE+CORRO under motion-correlated noise.
Proposed Experiments
The testbeds are Point-Robot and Sparse-Point-Robot, Half-Cheetah-Vel, Ant-Dir and Ant-Goal with densely resampled task parameters (10, 100, 1,000 and 10,000 training tasks), and Hopper-Param and Walker-Param for tasks that vary in dynamics, where splice error matters most. For manipulation we use Meta-World ML1 goal variation and ML45. In ML45, PEARL reaches only 30% success according to Meta-World. Metrics are test return or success under in-distribution context, OOD behavior context and the three noise types, together with probe accuracy for behavior and noise and GPU-hours.
Three outcomes would decide the question. The first is the scaling curve: our OOD-noisy return should improve with task count while per-update cost stays flat, and the random-negative ablation should plateau. The second is that under velocity-proportional noise our behavior probe should stay near chance while the probes for CSRO and CORRO rise. The third is that on dynamics-varying tasks, splice error should predict where our gains shrink. If the gains do not track splice error, the mechanism is not the one we claim.
References
- Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning
- HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces
- Probabilistic Label Trees for Extreme Multi-label Classification
- Context Shift Reduction for Offline Meta-Reinforcement Learning
- Behavior-Invariant Task Representation Learning with Transformer-based World Models for Offline Meta-Reinforcement Learning
- DROPO: Sim-to-real transfer with offline domain randomization
- On the use of Deep Autoencoders for Efficient Embedded Reinforcement Learning
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning