When Is World Feedback Transferable? A Convergence Gate in Contrastive Reinforcement Learning

RLxF @ ICML 2026 · OpenReview

TL;DR

Distilling knowledge from a large "teacher" model into a smaller "student" is a standard way to scale Reinforcement Learning (RL), but this process fails silently in Contrastive RL until the teacher crosses a sharp "convergence gate." This paper identifies that unlike scalar rewards, contrastive feedback requires a specific threshold of representation quality before it becomes transferable, a transition that can be predicted using simple, zero-cost behavioral diagnostics.

What Problem Does This Paper Address?

Reinforcement Learning usually relies on human feedback (RLHF), which is expensive and hard to scale. "World feedback"—automatic signals from the environment like "did I reach the goal?" or "what happens next?"—is a promising alternative.

The standard industry playbook is to train a massive, high-performance teacher model and then distill its behavior into a smaller, faster student model for deployment. However, the authors found that in Contrastive RL (where agents learn by discriminating between "correct" and "incorrect" future states), this distillation isn't a smooth process. A teacher might look like it's performing well, yet its students remain completely incompetent until a sudden "phase transition" occurs. Understanding when and why this "gate" opens is critical for anyone building autonomous agents or recommendation systems.

Methods

The researchers focused on Contrastive Reinforcement Learning (CRL), specifically using the InfoNCE objective. In this setup, the agent learns a representation that pulls "goal-consistent" states together and pushes "goal-inconsistent" states apart.

They compared this against:

  1. Scalar Reward RL (SAC): Traditional RL where the agent gets a single number (reward) for its actions.
  2. Temporal Contrastive RL (CURL): A self-supervised method that discriminates between states close together vs. far apart in time.

To investigate the "gate," they performed policy distillation, forcing a shallow student network to mimic the action distributions of a deep teacher network at various stages of the teacher's training. They also introduced a causal manipulation by varying the InfoNCE temperature ($\tau$), which controls how hard the discrimination task is for the encoder.

Datasets

The study utilized high-dimensional continuous control environments within the Brax simulator:

  • Ant Big Maze: A 29-dimensional observation space requiring complex navigation.
  • Humanoid: A 268-dimensional observation space with 17 actuators, representing a significant optimization challenge.
  • Arm Bin-Picking: Used for cross-task validation to see if the findings held up in robotic manipulation scenarios.

Key Insights and Ideas

  • The Gate is Real and Sharp: In CRL, student performance stays at zero for nearly 80% of the teacher's training time, then jumps to near-teacher levels almost overnight. This is a "step function" rather than a "smooth slope."
  • Form Matters: This gating phenomenon is unique to contrastive feedback. In traditional SAC (scalar rewards), distillation is smooth—better teachers always produce better students.
  • Representation is the Key: The gate isn't about the teacher's actions getting better; it's about the teacher's internal representations reaching a critical level of "alignment and uniformity." If the encoder is too shallow or unconverged, the gradients provided to the actor are essentially "noise," even if the teacher's own evaluation scores look decent.
  • Predictability: You don't need expensive probes to find the gate. The authors found that mean action magnitude ($|tanh(\mu)|$) is a highly accurate diagnostic. When the teacher's actions become "committed" (crossing a threshold of 0.63), the gate is likely open.

Key Results

  • Dramatic Amplification: Once the gate is crossed, distillation is incredibly powerful. On the Humanoid task, distillation turned a failing baseline (score of 6.4) into a walking policy that achieved 92% of the teacher's performance.
  • Efficiency: In scalar reward settings, they successfully transferred knowledge to an 11x smaller student while maintaining 96.6% of the teacher's reward.
  • Causal Control: By increasing the InfoNCE temperature (making the task "easier"), the researchers could shift the gate 13 epochs earlier in training, proving that contrastive difficulty is the primary driver of the bottleneck.
  • Diagnostic Accuracy: Their simple behavioral rule for gate detection achieved 90.5% accuracy across 21 different training conditions.

Limitations and Future Work

  • Methodological Scope: While the paper covers CRL, SAC, and CURL, it hasn't yet tested other world-feedback methods like BYOL-based RL or complex world-model objectives (e.g., Dreamer).
  • Sim-to-Real: All experiments were conducted in the Brax simulator. While Brax is standard for RL research, the dynamics of the "gate" in physical hardware or with real-world noisy sensors remain to be seen.
  • The "Scaffold" Mystery: The authors noted that if you remove the distillation signal after the student has learned, performance eventually degrades. They call the teacher a "persistent scaffold," but the exact reason why the student can't maintain peak performance independently is still an open question.
  • LLM Scaling: Applying these insights to "embodied LLMs" (large models trained on environment metrics) is a suggested next step for the field.

Summary updated

Abstract

World feedback, meaning measurable signals from agent-environment interaction such as goal reaching, future-state prediction, or system metrics, offers a path to scalable reinforcement learning that does not depend on human annotation. A standard recipe for deploying world-feedback agents is to amplify a strong teacher into smaller students via distillation. We show this recipe silently fails when the world-feedback signal is contrastive. In contrastive RL (CRL), where the teacher learns from an InfoNCE objective over future states, student performance is nearly constant across a wide range of partially trained teachers and then increases sharply once the teacher crosses a convergence gate: a sharp threshold in world-feedback representation quality. The gate is robust across three independent teachers, predictable from a zero-cost behavioral diagnostic ($\rho > 0.7$, $\mathrm{AUC} = 0.895$), and causally controlled by contrastive discrimination difficulty. Varying the InfoNCE temperature shifts the gate by 13 epochs. Once the gate closes, amplification is dramatic: on Humanoid, distillation transforms a near-zero baseline ($6.4$) into a walking policy at $92\%$ of teacher performance. By contrast, when the world-feedback signal is a scalar reward, SAC distillation is smooth and gate-free, transferring an $11\times$ smaller student to $96.6\%$ of teacher performance. The form of world feedback (dense contrastive versus scalar) can therefore determine whether amplification is smooth or gated. We close with operational guidance for world-feedback pipelines: cheap behavioral statistics on the teacher, such as mean action magnitude and output entropy, are stronger checkpoint-selection signals than validation loss, and contrastive temperature should be treated as an explicit knob over the gate's location. These results suggest that world feedback is not interchangeable. Its form governs whether a trained agent is a teacher one can reliably build on.
BibTeX
@inproceedings{2026when,
  title = {{When Is World Feedback Transferable? A Convergence Gate in Contrastive Reinforcement Learning}},
  booktitle = {RLxF @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=wFAiDIU4Ye},
}
Figure