When Are RL Hyperparameters Benign? A Study in Offline Goal-Conditioned RL

Jan Malte Töpperwien, Aditya Mohan, Marius LindauerarXiv 2026 · DOI · arXiv

TL;DR

Reinforcement Learning (RL) is notoriously finicky to tune, but this paper reveals that hyperparameter sensitivity isn't an inherent property of RL itself. By comparing bootstrapped (TD-learning) and non-bootstrapped (representation learning) methods in a controlled offline setting, the authors show that bootstrapping is a primary driver of brittleness, while non-bootstrapped objectives create "benign" landscapes that are remarkably robust to hyperparameter changes.

What Problem Does This Paper Address?

In the RL community, hyperparameter (HP) tuning is often viewed as a "black art." A configuration that works for one seed or one stage of training might fail miserably for another. This sensitivity is usually attributed to the "deadly triad" of RL: function approximation, bootstrapping, and off-policy learning.

However, in standard online RL, it is nearly impossible to tell if a failure is due to poor optimization (bad HPs) or poor exploration (the agent never saw the right data). This paper seeks to decouple these factors. By using offline goal-conditioned RL (GCRL), the authors can fix the data distribution and isolate how specific training mechanisms—specifically bootstrapping—amplify or mitigate hyperparameter sensitivity.

Methods

The authors employ a "phased training" protocol to study how the landscape of optimal hyperparameters evolves over time. They compare two representative algorithms:

  1. HIQL (Hierarchical Implicit Q-Learning): A state-of-the-art method representing the bootstrapped TD-learning paradigm. It relies on recursive value estimates (targets that depend on the model's own current predictions).
  2. QRL (Quasimetric RL): A method representing the representation learning paradigm. It learns a latent distance metric between states without using Bellman backups or bootstrapping.

To analyze these, they use:

  • Response Surfaces: Visualizing performance across the learning-rate/discount-factor plane.
  • fANOVA: A statistical tool to identify which specific hyperparameters contribute most to performance variance.
  • Inter-goal Gradient Alignment: A novel diagnostic that measures whether gradients from different relabeled goals reinforce or conflict with each other during the critic update.

Datasets

The study utilizes the OGBench suite, focusing on complex navigation tasks:

  • antmaze-large-v0 and antmaze-medium-v0
  • pointmaze-medium-v0
  • humanoidmaze-medium-v0
  • antmaze-teleport-v0

Crucially, they manipulate data quality by mixing "expert" trajectories with "exploratory" (noisy) data. They test both stationary datasets (fixed quality) and scheduled datasets (where data quality improves over training phases) to simulate the non-stationarity of online RL in a controlled way.

Key Insights and Ideas

  • Bootstrapping is the "Sensitivity Multiplier": The most striking finding is the divergence between the two methods. HIQL (bootstrapped) has "sharp" optima that drift significantly as training progresses. QRL (non-bootstrapped) maintains broad, stable "benign" regions where many different HP settings work equally well.
  • The 20% Expert Threshold: Once a dataset contains even a modest amount of expert data (~20%), non-bootstrapped representation learning becomes highly robust. In contrast, bootstrapped methods remain sensitive even with high-quality data.
  • Destructive Gradient Interference: The authors link HIQL’s sensitivity to "inter-goal gradient interference." In bootstrapped objectives, the gradients for different goals often fight each other (negative cosine similarity). This conflict shrinks the "viable" hyperparameter space, making the optimizer walk a tightrope.
  • Offline RL is Generally More Robust: Across the board, offline RL landscapes are more benign than what is typically reported for online RL. This suggests that a huge portion of RL's famous "brittleness" actually stems from the chaotic feedback loop between exploration and optimization.

Key Results

  • Landscape Stability: QRL's near-optimal hyperparameter regions (the top 10% of configurations) remain largely anchored in the same place across training phases. HIQL’s optimal regions move significantly, meaning a "good" learning rate at Phase 1 might be "bad" by Phase 4.
  • HP Importance: fANOVA results show that QRL is largely dominated by a single hyperparameter (learning rate). HIQL, however, has a much more distributed importance profile, meaning users have to tune a wider array of "knobs" (like target network update rates and hierarchy-specific HPs) to get it to work.
  • $\epsilon$-optimality Mass: QRL consistently exhibits a higher "mass" of near-optimal configurations. In some expert data settings, nearly 50% of all tested HP configurations for QRL reached near-optimal performance, compared to much lower fractions for HIQL.

Limitations and Future Work

  • Algorithmic Scope: The study focuses on two specific (though representative) algorithms. It remains to be seen if these findings hold for other popular offline methods like CQL or IQL.
  • The "Efficiency vs. Robustness" Trade-off: While non-bootstrapped methods like QRL are more robust to HPs, bootstrapping is often more sample-efficient in specific regimes. The authors suggest the next frontier is designing objectives that capture the efficiency of TD-learning without its inherent instability.
  • Bridging back to Online RL: While the offline setting provided clarity, the ultimate goal is to apply these "benign" objective designs back to online agents to reduce the need for massive compute-heavy hyperparameter sweeps.

Summary updated

Abstract

Hyperparameter sensitivity in Deep Reinforcement Learning (RL) is often accepted as unavoidable. However, it remains unclear whether it is intrinsic to the RL problem or exacerbated by specific training mechanisms. We investigate this question in offline goal-conditioned RL, where data distributions are fixed, and non-stationarity can be explicitly controlled via scheduled shifts in data quality. Additionally, we study varying data qualities under both stationary and non-stationary regimes, and cover two representative algorithms: HIQL (bootstrapped TD-learning) and QRL (quasimetric representation learning). Overall, we observe substantially greater robustness to changes in hyperparameter configurations than commonly reported for online RL, even under controlled non-stationarity. Once modest expert data is present ($\approx$ 20\%), QRL maintains broad, stable near-optimal regions, while HIQL exhibits sharp optima that drift significantly across training phases. To explain this divergence, we introduce an inter-goal gradient alignment diagnostic. We find that bootstrapped objectives exhibit stronger destructive gradient interference, which coincides directly with hyperparameter sensitivity. These results suggest that high sensitivity to changes in hyperparameter configurations during training is not inevitable in RL, but is amplified by the dynamics of bootstrapping, offering a pathway toward more robust algorithmic objective design.
BibTeX
@misc{topperwien2026when,
  author = {Jan Malte Töpperwien and Aditya Mohan and Marius Lindauer},
  title = {{When Are RL Hyperparameters Benign? A Study in Offline Goal-Conditioned RL}},
  year = {2026},
  eprint = {2602.05459},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.05459},
}
Figure