Reinforcement learning agents often settle on a mediocre behavior early and stop improving. This is premature convergence: the agent stops exploring because a nearby, mediocre solution already looks good. In long-horizon tasks we argue the usual cause is not too little randomness. It is a credit-assignment blind spot. A value estimator that looks one step ahead cannot see past a stretch of low reward to a better outcome on the far side, so every small change of behavior looks worse than the current rut. Value estimators that score multi-step action sequences, called action "chunks", can see across such gaps. They can also be wrong in ways that trap the agent somewhere else.
We propose to train value estimators at several lookahead lengths at once. The agent then reads the disagreement between them, state by state, and uses it to choose how far ahead to commit and how much the one-step estimator should defer to the longer ones. Long chunks are used where a confident long-horizon estimate sees value that the one-step estimate misses. Short reactive control is used where the long-horizon estimate is uncertain. The claim is that the length of temporal commitment, applied to both behavior and the value backup, is the right escape knob. Cross-horizon disagreement gives a way to turn that knob that needs no estimate of the optimal return.
Core Claim
Existing fixes for premature convergence act at the single-step level. Goal Achievement Guided Exploration (GAGE) keeps a floor under the policy's action noise while measured task progress is low. That requires an estimate of the optimal return, which its authors' own setting shows is hard to define in open-ended tasks. Temporal-abstraction methods fix the horizon. Reinforcement Learning with Action Chunking (Q-chunking, QC) learns a critic over fixed-length chunks. Chunk-Guided Q-Learning (CGQ) distills a one-step critic toward a fixed-$h$ chunked critic with a static weight. The CGQ authors report that this weight must be hand-tuned, and that a noisy chunked critic in AntMaze drags the policy into suboptimal behavior. That is a second kind of entrapment, one the chunking itself causes.
Our claim is that both traps can be detected and escaped with one signal. Let $Q_h(s,\cdot)$ be an ensemble of critics, each estimating the return of executing an $h$-step chunk from state $s$ and then following the policy, for $h \in \{1, 5, 10, 25\}$. Because every critic estimates the same quantity, the return, their values are directly comparable. Let $\mu_h(s)$ and $\sigma_h(s)$ be the ensemble mean and standard deviation of the best chunk's value at horizon $h$. The agent commits to the horizon with the highest pessimistic estimate:
$$h^*(s) = \arg\max_h \; \big[\mu_h(s) - \kappa\,\sigma_h(s)\big],$$
where $\kappa$ sets how much uncertainty is penalized. The one-step critic is pulled toward a longer horizon only when that horizon's pessimistic estimate exceeds its own:
$$\beta_h(s) = \mathrm{sigmoid}\!\left(\frac{\mu_h(s) - \kappa\,\sigma_h(s) - \mu_1(s)}{\tau}\right),$$
with temperature $\tau$. The distillation is one-directional, so long-horizon critics are never regressed toward the one-step critic. We borrow this asymmetry from Multiple-Frequencies Population-Based Training, where letting fast, greedy agents overwrite slow ones caused population-wide collapse. Our version moves the idea from the population level into the value backup.
The design makes a testable prediction. Escapes from plateaus should be preceded by states where a long horizon wins the selection rule. In tasks where long-horizon critics are unreliable, the rule should fall back to short horizons and avoid the AntMaze failure that CGQ reports.
Research Program
The method builds on QC's flow-matching chunk policy, which already produces a full chunk. We execute only its first $h^*(s)$ actions and then replan, as Mixture of Horizons in Action Chunking does when it executes action prefixes based on agreement across horizons. That work is imitation-only, with no value backups. We replace its learned gate with a value-based rule that can respond to reward. Each horizon's critic is trained with its own window-aligned target, in the spirit of Implicit Action Chunking for Smooth Continuous Control. Off-policy concerns stay minimal because each chunked critic conditions on the full action sequence, as in QC. Learning Routines for Effective Off-Policy Reinforcement Learning warns that very long horizons degrade critic accuracy. The ensemble spread $\sigma_h$ is how we expect to detect that degradation per state, rather than avoid long horizons globally.
The work proceeds in three stages. The first is a diagnostic study on existing agents. On OGBench tasks where single-step learners stall, we log the cross-horizon gap over training to test whether stalls coincide with a large, confident $\mu_h - \mu_1$. That would mean the one-step critic is blind to value the long critic sees. The second stage builds and tests the full method in the offline-to-online setting QC uses. The third stage ablates the method so that each ingredient's contribution is visible separately.
Baselines
The naive reference is FQL, the single-step flow Q-learning agent. With added exploration noise it represents "do nothing about horizons". On OGBench manipulation it averages 38%, as reported in the CGQ paper.
The fixed-horizon reference is QC. It reports about 86% aggregate success across five OGBench domains and 74% on cube-quadruple, where most one-step TD baselines reach 0% (Reinforcement Learning with Action Chunking).
The strongest competitor is CGQ, because it already combines a chunked critic with a reactive one-step critic, which is the design we modify. It reports a 76% OGBench manipulation average, against DEAS at 71%, and 94% on cube-triple, where single-step methods reach 10% (Chunk-Guided Q-Learning. The file gives no CGQ number on AntMaze, so no reported result was found on Lacuna for CGQ on AntMaze. We will rerun it. Beating CGQ means a higher manipulation average than its rerun score at matched compute. It also means a clear gain on locomotion tasks where it struggles, without losing on cube-triple.
Fairness rules apply to every method. All methods get the same datasets, the same online interaction budget, and the same total critic parameter count; CGQ and QC receive ensembles of equal total size. Every method gets an equal hyperparameter-search budget. For CGQ this includes its $\beta$; for QC and CGQ it includes $h$.
Four ablations isolate the key ingredients. The first keeps adaptive behavior horizons but uses a single fixed critic horizon, which tests whether the backup horizon matters beyond the behavior horizon. The second keeps multi-horizon critics but uses a constant $\beta$ in place of the gated $\beta_h(s)$. The third sets $\kappa = 0$, removing the uncertainty penalty. The fourth samples $h$ at random per decision, which controls for the effect of mere horizon diversity.
Why not just…?
Why not just keep more action noise, GAGE-style, or use temporally correlated noise? This is the most serious alternative. Correlated noise gives coherent exploration without any chunk machinery. Neither noise scheme changes how far value propagates per update, however, and our hypothesis is that the trap lies in the backup, not only in the data. QC's 0% for one-step TD on cube-quadruple is suggestive but does not settle the question. We will run a correlated-noise baseline with a GAGE-style variance floor. The first ablation (adaptive behavior horizon, fixed one-step critic) is the decisive test. If it matches the full method, our claim about the critic is wrong, and the contribution reduces to an exploration schedule.
Why not just use novelty counts, as SNAP does? State-Novelty Guided Action Persistence repeats actions more often in rarely visited states. Novelty does not track entrapment, though. An agent stuck in a familiar rut sees low novelty and switches to short actions, which is exactly backwards. We will implement count-based persistence on top of QC and check whether it helps on stalled seeds, the runs that have plateaued.
Why not just tune the best fixed $h$ per task, or run population-based training over $h$? A per-task sweep is a real threat, so we report an oracle that picks the best fixed $h$ for each task. Our method must beat that oracle on tasks that mix long navigation with precise contact, such as Scene and cube-quadruple. If it only matches the oracle, the result is honest but modest: tuning-free horizon choice. Population-based training is a heavier version of the same per-task tuning. MF-PBT needed around 1B Brax steps, and standard PBT itself collapsed (16,171 IQM on Humanoid versus 17,713 for random search), so we treat a fixed-$h$ sweep as the affordable proxy.
Decisive Experiments
All experiments use OGBench in the offline-to-online protocol: cube-triple, cube-quadruple, Scene, and Puzzle for manipulation, and AntMaze and HumanoidMaze for locomotion. We include giant HumanoidMaze and the hardest Puzzle levels, where every tested offline baseline scores 0%. We use at least ten seeds per task.
We report three groups of metrics. The first is final success rate. The second is the fraction of seeds stuck below 10% success after half the budget, our direct measure of entrapment, together with mean plateau length. The third covers the horizon chosen and the $\beta_h$ value along trajectories.
Three checks are decisive. First, on stalled seeds of the fixed-horizon baselines, our method should show long-horizon selections concentrated just before success rises. Second, on AntMaze, $\beta_h$ should fall where $\sigma_h$ is high, and success should exceed rerun CGQ. Third, on cube-triple, success should stay near CGQ's reported 94%, showing that adaptivity does not cost precision.
If correlated noise or the fixed-critic ablation closes the gap, we will report that the horizon of commitment matters for exploration but not for credit assignment.
References
- Goal Achievement Guided Exploration: Mitigating Premature Convergence in Learning Robot Control
- Reinforcement Learning with Action Chunking
- Chunk-Guided Q-Learning
- Multiple-Frequencies Population-Based Training
- Mixture of Horizons in Action Chunking
- Implicit Action Chunking for Smooth Continuous Control
- Learning Routines for Effective Off-Policy Reinforcement Learning
- State-novelty guided action persistence in deep reinforcement learning
- OGBench: Benchmarking Offline Goal-Conditioned RL