Reflect-then-Correct: Rebalancing Task Optimization for Generalizable Meta-Reinforcement Learning via Distributional Value Error Reduction

Min Wang, Xin Li, Ye He, Yonggang ZhangICML 2026 · OpenReview

Meta-Reinforcement Learning (Meta-RL) aims to create agents capable of "learning how to learn," allowing them to quickly adapt to new tasks based on limited experience. While Meta-RL excels at parametric tasks—where the goal changes only slightly (e.g., a simulated robot running at different target speeds)—it struggles significantly in non-parametric settings. In these environments, tasks are semantically distinct, such as switching from opening a door to playing basketball.

To address this, researchers have introduced Reflect-then-Correct (RTC), a novel framework designed to help Meta-RL agents generalize across highly diverse tasks. By leveraging Sinkhorn divergence to handle varying reward scales and a recursive error-correction model to prioritize informative training data, RTC significantly outperforms existing methods on challenging benchmarks.

The Challenge: Gradient Interference and Reward Scales

The core obstacle in diverse, non-parametric settings is gradient interference. Because fundamentally different tasks yield vastly different reward scales, the optimization process tends to over-prioritize tasks with large return magnitudes. Consequently, the agent fails to learn more difficult, low-reward tasks.

Existing categorical solutions attempt to solve this by "binning" rewards to normalize scales. However, these rigid discretization methods often introduce quantization errors and lack the flexibility needed for precise value estimation.

The RTC Framework: Reflect and Correct

RTC abandons rigid scalar predictions and fixed bins, instead breaking the learning process into two adaptive phases:

1. Sinkhorn Distributional Learning (The "Reflect" Phase)

Rather than predicting a single scalar value, RTC models the "meta-value" as a distribution of adaptive "floating particles." It utilizes Sinkhorn divergence—a computationally efficient variant of Optimal Transport—to align these distributions. This geometry-aware approach allows the agent to identify structural similarities between tasks, rendering the model invariant to the absolute scale of rewards and leveling the playing field for tasks of varying difficulty.

2. Recursive Error Correction (The "Correct" Phase)

While Sinkhorn divergence provides a smoother learning signal, its sampling-based nature introduces statistical bias. RTC overcomes this by "reflecting" on Bellman inconsistencies, tracking how errors accumulate over time through a recursive model. It then "corrects" the optimization process by applying adaptive importance weights. This ensures the agent focuses its learning efforts on the transitions most critical for accurate meta-value estimation, rather than simply chasing the largest rewards.

Key Insights and Mechanisms

  • Latent Specialization: Visualizations of the learned task representations reveal that RTC creates much cleaner, more distinctly separated clusters for different task categories compared to baseline methods. This suggests the model successfully captures the underlying "essence" of each task.
  • Loss Landscape Smoothing: RTC produces a notably flatter and smoother loss surface. In deep learning, a flatter loss landscape is a strong indicator of improved generalization and robustness.
  • Bias-Variance Tradeoff: The recursive error model acts as a clever "de-biasing" mechanism, allowing the framework to enjoy the geometric benefits of Sinkhorn divergence without suffering from its inherent sampling bias.

Experimental Results on Meta-World

The researchers evaluated RTC on two highly challenging benchmarks from Meta-World, where test tasks feature fundamentally different dynamics and objectives than the training tasks:

  • ML-10: 10 diverse robotic manipulation tasks for training, 5 unseen for testing.
  • ML-45: 45 training tasks, 5 unseen for testing.

RTC demonstrated superior performance across the board:

  • Higher Success Rates: On the ML-10 test set, RTC achieved a 30.42% success rate, outperforming the next best baseline (variBAD at 28.39%) and significantly beating the recent AMAGO-2 architecture (10.89%).
  • Better Returns: RTC achieved a test return of 1394.75 on ML-10, nearly 30% higher than its closest competitor.
  • Robustness at Scale: In the highly diverse ML-45 setting, RTC maintained its lead with a 28.66% test success rate, whereas competing methods saw sharp performance drops as task diversity increased.

Limitations and Future Directions

While RTC represents a significant step forward for multi-task Meta-RL, challenges remain. Implementing Sinkhorn iterations and maintaining an auxiliary recursive error model introduces computational overhead compared to simpler scalar regression. Furthermore, while RTC achieves state-of-the-art results, a ~30% success rate on entirely unseen, non-parametric tasks highlights that this remains an open and exceptionally difficult problem.

Future work may explore integrating RTC with transformer-based architectures to further boost generalization, as well as addressing the critical safety considerations required before deploying such adaptive systems in real-world robotics.

Summary updated

Abstract

Meta-Reinforcement Learning (Meta-RL) faces significant challenges in non-parametric settings, where vastly different return scales across diverse tasks cause severe gradient interference. Existing categorical solutions attempt to normalize these scales but often fail due to rigid discretization and quantization errors. To address this, we propose Reflect-then-Correct (RTC), a framework that models meta-values using Sinkhorn divergence. By treating distributions as adaptive floating particles, RTC achieves a geometry-aware alignment of distinct meta-task structures. However, while Sinkhorn updates harmonize gradients, they introduce statistical bias via sampling estimation. RTC overcomes this issue by "reflecting'' on the temporal accumulation of Bellman inconsistencies through a recursive error model and "correcting'' the optimization via adaptive importance weights, which prioritize more accurate transitions for meta-value estimation. We provide theoretical guarantees for this reweighting strategy and demonstrate that RTC outperforms existing baselines on the challenging Meta-World ML-10 and ML-45 benchmarks.
BibTeX
@inproceedings{wang2026reflect,
  author = {Min Wang and Xin Li and Ye He and Mingzhong Wang and Yonggang Zhang},
  title = {{Reflect-then-Correct: Rebalancing Task Optimization for Generalizable Meta-Reinforcement Learning via Distributional Value Error Reduction}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=AJMrn5qYih},
}
Figure