Robots that plan with a learned world model usually compress camera images into a small vector, a latent state, and then search for a good action sequence by imagining rollouts in that compressed space. The search methods they use are greedy. They include the cross-entropy method, which repeatedly samples action sequences and refits a Gaussian to the best few; hill climbing on a single candidate; and local second-order trajectory solvers. All of them tend to commit early to one way of solving the task. A car that should brake before a corner never finds that out, because braking looks worse over a short horizon. A manipulator settles on the first approach to a tool it happens to find. A maze agent picks a corridor that turns out to be a dead end.
We propose to keep several distinct candidate solutions alive while the planner runs. Each candidate is a "niche" with its own small optimizer, and niches carry over from one control step to the next. They push each other apart only when they are heading to the same place. The key choice is how "same place" is measured. We do not compare action sequences or raw latent vectors. We compare the latent waypoints of the candidate trajectories in a learned reachability embedding, a space where distance approximates the number of steps needed to get from one state to another. Two plans with very different action sequences that end up in the same corridor count as one route. Two plans that look alike at first but split around an obstacle count as two.
Core Claim
Premature convergence in latent planners is mostly route collapse: the candidates converge onto one route through state space, not merely onto one action sequence. It can be prevented cheaply if diversity is measured by reachability between imagined states. Our concrete claim has three parts. First, a planner that keeps reachability-separated niches across control steps will solve tasks with several competing routes more often than the planners in the file. These include single-candidate evolutionary search (EPLS), local collocation (LatCo), and improved cross-entropy search (iCEM). Second, it should do so at an equal budget of model rollouts. Third, the same planner with diversity measured in action space or raw latent space should lose most of that gain.
The failures we target are documented in the file. EPLS runs one-candidate hill climbing inside a VAE and recurrent world model on CarRacing. It inherits track biases, failing on right turns and refusing to brake before corners, and its performance degrades beyond a 20-step horizon. LatCo optimizes latent states and actions jointly and temporarily relaxes dynamics so the plan can "teleport" toward high reward. Its authors still report that the Levenberg–Marquardt solver falls into local minima, and the solver commits to whichever branch its initialization favors.
Research Program
The planner has four parts.
The first is the world model. We use a standard recurrent latent world model of the kind used by EPLS and LatCo, trained on the agent's own replay data.
The second is a reachability encoder $\phi$. It maps latent states into a space where Euclidean distance approximates temporal distance. It is trained on replay trajectories by regressing embedding distances onto step counts between pairs of states. The method is borrowed directly from SCoTS, which builds such an embedding to stitch offline trajectory segments for diffusion planners. SCoTS uses the embedding only to augment an offline dataset before training. We use it online, inside the planner, as a distance between plans.
The third part is the set of niches. Each niche $i$ holds a candidate latent trajectory $\tau_i$ and runs either a LatCo-style collocation update or a small cross-entropy update. We test both, so the claim is not tied to one inner optimizer. Niches interact through a repulsion term that is used when updating them, but not when choosing the action to execute:
$$ S(\tau_i) = J(\tau_i) - \lambda \sum_{j \neq i} \exp\!\left(-\frac{d(\tau_i,\tau_j)^2}{2\sigma^2}\right), \qquad d(\tau_i,\tau_j) = \frac{1}{M}\sum_{m=1}^{M} \big\|\phi(z^i_{t_m}) - \phi(z^j_{t_m})\big\|. $$
Here $J$ is the predicted return under the world model, $z^i_{t_m}$ is the $m$-th of $M$ evenly spaced latent waypoints along $\tau_i$, $\sigma$ is a reachability scale measured in steps, and $\lambda$ sets the repulsion strength. The Gaussian gate follows DPSO, which pushes a particle's best point away from the swarm's best only when the two are close. DPSO's kernel acts on raw parameter vectors and was tested only on static benchmark functions. Ours acts on imagined routes inside a dynamics model.
Two further rules address limitations reported elsewhere in the file. DPSO-style repulsion hurts precision on landscapes with a single optimum, and DynAMO shows that forcing spread lowers median batch quality. To guard against both, any two niches closer than $\sigma$ are merged, and the freed budget goes to the leading niche. When the task really has only one route, the planner should therefore become an ordinary greedy optimizer. The second rule borrows from MF-PBT, which protects slow-maturing agents in population-based training from being overwritten by greedy clones. In our planner, the leading niche may overwrite a trailing niche only if the trailing niche's return has lagged for $K$ consecutive control steps. Niches are warm-started by shifting them one step forward in time, so a route whose payoff arrives after the planning horizon, such as braking now to take the corner faster, can survive long enough for its payoff to show.
The fourth part is the executed action. It comes from the niche with the highest predicted return, so repulsion never directly degrades the action the robot takes.
The analysis has two parts. We will measure route diversity directly by counting the distinct maze corridors visited by live niches over an episode, and relate it to success. We will also test whether the reachability encoder gives a second benefit. A relaxed collocation step whose consecutive waypoints are far apart in $\phi$ is a candidate jump through a "hole" in the latent space, meaning a region the model never learned. We will check whether penalizing such jumps reduces decoded infeasible plans. That penalty is a cheaper substitute for DynAMO's adversarial critic, which costs about 180% extra runtime and is impractical for real-time control.
Baselines
The naive reference point is the base optimizer with one population and no niching. That means plain cross-entropy planning in the latent model for the shooting variant, and plain LatCo for the collocation variant, each tuned to its best. This tells us whether any diversity mechanism helps at all.
The strongest existing competitor depends on the task. On sparse-reward MetaWorld it is LatCo, which Rybkin et al. report at a 91% success rate on Reaching (PlaNet's shooting planner: 20%) and up to 37% success on Hammer and Thermos, where shooting baselines get 0%. It is the strongest competitor because it already escapes some local traps through dynamics relaxation. On CarRacing-v0 the reference is EPLS, reported at 708 ± 195 after five iterative cycles, against 765 ± 102 for a supervised expert model (Olesen and Nguyen). For sampling-based search we include iCEM, which adds temporally correlated noise and keeps elite candidates across iterations. The file reports its gains only as relative figures on Gym and DAPG tasks: 2.7 to 21.9 times fewer samples, and 90% success on Relocate. No reported iCEM result on our benchmarks was found on Lacuna. No reported result was found on Lacuna for online latent planners on OGBench mazes either; SCoTS's 100% on PointMaze Giant comes from an offline diffusion planner trained on stitched data, so it is a different setting and serves only as context.
Fairness rules apply to all methods. Every method plans with the same trained world model and the same number of imagined rollouts per control step. For collocation the matched quantity is solver iterations times trajectories. Each method gets the same hyperparameter search budget. Wall-clock time is reported separately.
Two ablations isolate the key ingredient. In the first, the reachability distance $d$ is replaced by Euclidean distance between action sequences. In the second, it is replaced by Euclidean distance between raw latent states. A third ablation removes persistence across control steps by reinitializing niches at every step.
We beat a baseline if we raise success or return by more than the seed-to-seed spread across 10 seeds on multi-route tasks, and lose no more than that spread on single-route tasks.
Why not just…?
Why not just give the cross-entropy method more samples, or use iCEM? iCEM makes sampling far more efficient. Its Gaussian refit still shrinks toward a single mode, and LatCo's results show shooting planners stuck at 0% on tool use however they are tuned. We test this directly: iCEM is given up to four times our rollout budget. If it matches us at that budget, our contribution reduces to an efficiency gain, and we will report it as such.
Why not just run several independent restarts of LatCo or the cross-entropy method? This is the most serious threat. Independent populations already produce some diversity, and on easy landscapes they may find the same routes. Our bet is that independent restarts waste budget by repeatedly finding the same route, and that they lose long-payoff routes at every control step because nothing protects those routes over time. The equal-budget comparison against $K$ independent restarts, both reinitialized and warm-started, is the experiment that decides this. If warm-started independent restarts match the full method, we will report that repulsion is unnecessary and that the contribution is persistence alone.
Why not just add diversity in action space, through noise, entropy bonuses, or DPSO on action sequences? GAGE shows that enforcing action variance helps model-free policy learning. Over long horizons, though, independent action noise mostly cancels out, which is iCEM's own argument for correlated noise, and very different action sequences can lead to the same latent state. The action-space-distance ablation tests this alternative precisely.
Why not just use population-based training? MF-PBT reports that standard PBT collapses and underperforms random search on Humanoid (an interquartile-mean reward of 16,171 against 17,713). More basically, PBT works across training runs and cannot react within an episode to a dead-end corridor.
Proposed Experiments
There are three testbeds.
The first is OGBench PointMaze and AntMaze in the Medium, Large and Giant layouts, run as an online planning problem with a learned world model. These mazes have several known routes, which makes route counting possible and makes them the clearest test of route collapse.
The second is CarRacing-v0, which revisits EPLS's documented failures. Alongside episode return, we will measure right-turn failure rates and pre-corner braking, and sweep the planning horizon from 10 to 40 steps. The question is whether keeping several niches pushes back the degradation EPLS saw beyond 20 steps, or whether that degradation comes purely from model error.
The third is sparse-reward MetaWorld Reaching, Hammer and Thermos against LatCo, plus dense-reward MetaWorld tasks as single-route controls where we expect no gain.
Four decisive checks follow from the claim. First, on mazes, success must rise together with the number of distinct routes held by live niches, and the reachability-distance version must beat the action-space and raw-latent ablations. If the raw-latent ablation matches it, the reachability encoder is not needed. Second, the full method must beat warm-started independent restarts at an equal budget. Third, on dense single-route tasks, merging must keep any loss within seed noise, which tests whether we avoid the precision cost DPSO and DynAMO pay. Fourth, Hammer and Thermos success must exceed LatCo's reported 37% when our method uses the LatCo solver inside each niche. Failing the second check would mean the idea is persistence, not diversity. Failing the first would mean route collapse is the wrong diagnosis.
References
- Evolutionary Planning in Latent Space
- Model-Based Reinforcement Learning via Latent-Space Collocation
- Sample-efficient Cross-Entropy Method for Real-time Planning
- State-Covering Trajectory Stitching for Diffusion Planners
- Divergence-Guided Particle Swarm Optimization
- Diversity By Design: Leveraging Distribution Matching for Offline Model-Based Optimization
- Multiple-Frequencies Population-Based Training
- Goal Achievement Guided Exploration: Mitigating Premature Convergence in Learning Robot Control