Tethered Exploration: Using the Go-Phase Policy as Its Own Safety Net

Agents that learn by trial and error waste most of their experience wandering near where they started. One effective fix, called two-phase or "Go-Explore"-style exploration, splits each episode in two. First a goal-reaching policy drives the agent quickly across territory it already knows to the edge of that territory. Then a random or curiosity-driven policy pokes into the unknown. The trouble is that the edge of the known world is exactly where hazards the agent has not yet mapped tend to sit. Current two-phase methods take unconstrained random actions there. This proposal makes the random excursion safe without an outside safety controller. The agent may step into the unknown only while its own go-phase policy could still bring it back, with low predicted cost, to territory it has mastered. When that stops being true, control snaps back to the go-phase policy, like an animal on a leash.

In two sentences: the goal-reaching policy that two-phase exploration already learns is also a safe fallback controller, as long as we can predict how hard it is to get back. We learn a direction-aware "time to return" estimate and a "cost to return" estimate, and allow exploratory actions only when both stay within budget. The result should keep the sample-efficiency gains of frontier exploration while cutting the number and length of safety violations during training. The gain should be largest in environments with one-way transitions, where symmetric distance estimates wrongly report that a dead end is close to home.

Core Claim

Two-phase methods such as LEAF and CE² learn a goal-conditioned policy and a reachability measure over a mastered region. They then throw that structure away the moment the explore phase begins. LEAF uses a latent network that predicts whether one state is reachable from another within k steps and explores stochastically past the frontier. CE² fits Gaussian mixtures to a world model's latent space and explores undirected from low-density cluster edges. The research file notes that CE²'s edge selector tends to pick goals next to hazards, because the edge of explored space is often where the agent was previously stopped.

Safe exploration, meanwhile, often needs a trusted fallback. SOOPER switches to a conservative prior policy whenever predicted cost under a probabilistic world model exceeds the budget. It reports a physical race car whose learned policy was twice as good as its safe prior with zero training collisions. However, SOOPER requires that prior to exist in advance and relies on resets. Safe Stochastic Explorer requires a high-probability safe path back to a home base before each action. It does this with Gaussian processes whose cubic cost limits scale.

Our claim is that the go-phase policy can serve as a self-built SOOPER prior, and that the return condition from Safe Stochastic Explorer can be checked cheaply with learned temporal distances. For this to be sound, the distance must be asymmetric. TLDR learns representations where Euclidean distance equals the minimum number of steps between states, which assumes going from A to B costs the same as B to A. That assumption is harmless for coverage but dangerous for safety. A ledge you can drop off in one step is "one step away" in both directions under a symmetric metric, yet it is unrecoverable.

Mechanism

Let $M$ be the mastered set: states from which the go-phase policy $\pi_g$ reaches commanded goals with empirical success above a threshold, measured from real rollouts rather than inferred from the latent space. We learn two quantities:

  • $D_\theta(s, m)$, an asymmetric (quasimetric) estimate of the steps $\pi_g$ needs to travel from state $s$ to $m \in M$.
  • $C_\phi(s)$, the expected safety cost incurred by $\pi_g$ while returning from $s$ to the nearest mastered state.

Both are trained on real and world-model-imagined rollouts of $\pi_g$. During the explore phase, a proposed action $a$ with predicted next state $s'$ is permitted only if

$$\min_{m \in M} D_\theta(s', m) \le L \quad \text{and} \quad c_{\text{spent}} + \hat c(s,a) + C_\phi(s') \le \epsilon_{\text{ep}} .$$

Here $L$ is the leash length in steps, $c_{\text{spent}}$ is the cost already incurred this episode, $\hat c(s,a)$ is the world model's predicted immediate cost, and $\epsilon_{\text{ep}}$ is the per-episode budget from the constrained formulation. If the check fails, control returns to $\pi_g$ commanded toward the nearest mastered state.

The leash $L$ is not hand-tuned. It grows when the return predictions have been validated, meaning realized return times fall within their predicted confidence band, and it shrinks after a prediction error. This replaces the fixed thresholds the file flags as an open problem in LEAF, KEA, and CE² with a quantity that is checked against outcomes.

Frontier goals are chosen from the boundary of $M$ by expected new coverage per unit of return risk, not by novelty alone. This pushes exploration away from hazard-adjacent edges that are cheap to reach but expensive to leave.

This design also changes how the safety model gets its training data. FDPI showed that as agents become safe, the model that estimates which states are safe loses boundary data, and late-training cost spikes follow. FDPI fixes this with a dual policy that deliberately violates constraints, which is unusable on hardware. Tethered excursions instead produce a steady stream of near-boundary states that the agent can recover from, the "near misses." These come from the leash edge, which is by construction the place where $D_\theta$ and $C_\phi$ are most uncertain. Whether near-miss data suffices to keep the safety model accurate, without real violations, is a central empirical question of the proposal.

Research Program

We build on a DreamerV3 world model with a cost head, so the system is directly comparable to SafeDreamer and to CE², which uses the same backbone. The quasimetric distance head and the cost-to-return critic are trained in imagination and corrected with real return rollouts.

The work has three parts. The first shows that asymmetric distance matters: we measure return-time prediction error, and false "safe" classifications near one-way transitions, for symmetric (TLDR-style) and asymmetric heads. The second is the full tethered system compared against the baselines below. The third is an analysis of the safety paradox. Following FDPI's diagnostic, we track the fraction of boundary-relevant samples in the replay buffer and the calibration error of $C_\phi$ over training.

The theory result that safe exploration incurs nearly no additional sample complexity for reward-free RL motivates the target. We expect state coverage per environment step to stay close to unconstrained CE² rather than dropping to monolithic safe-RL levels.

Baselines

The naive references are PPO-Lag and SafeDreamer run as monolithic learners with no two-phase structure. Safety-Gymnasium reports that PPO-Lag achieves a 98% reduction in violations over PPO with a 45% drop in reward on velocity tasks. SafeDreamer reports a 94.3% cost reduction relative to other model-based methods and a 100% arrival rate at average episode cost 2.2 versus 14.1 for PPO reward shaping on MetaDrive. SafeDreamer is the strongest existing safe learner in the file and shares our backbone, so it is the main safety competitor. We expect it to be safe but slow to discover sparse goals in large mazes.

The strongest exploration competitor is unconstrained CE², which the file reports exceeding 90% success on 3-Block Stacking where MEGA and PEG stayed below 40%. We treat it as a coverage upper bound that ignores cost.

The most dangerous baseline is the obvious combination we call Safe-CE². It keeps CE²'s go/explore split but trains the explore policy with SafeDreamer's PID-Lagrangian cost penalty.

We also run SOOPER with the go-phase policy plugged in as its prior. This tests whether our distance-based tether adds anything over SOOPER's predicted-cost fallback given the same fallback controller.

Ablations isolate the key ingredient in three ways. We replace the asymmetric distance with a symmetric one. We drop the cost-to-return term, keeping only the distance leash. We fix $L$ rather than adapting it.

All methods share the DreamerV3 backbone where applicable, the same environment-step budget, the same cost budget, and an equal hyperparameter search of the same number of configurations per method. We report five or more seeds. Beating Safe-CE² means two things: halving cumulative training cost and the Expected Maximum Consecutive Cost steps (EMCC) metric, while matching its success rate and coverage within seed noise. Beating SafeDreamer means reaching its final cost level with substantially higher sparse-goal success at equal steps. We found no reported results for any baseline on the hazard-augmented maze variants below, so all numbers there will be produced under this shared protocol.

Why not just…?

Why not just run CE² with a Lagrangian cost penalty on the explore policy? This is the real threat. In environments where hazards are reversible, Safe-CE² may match us. A Lagrangian penalty constrains expected cost over training rather than acting per step. It is known to oscillate between recklessness and freezing (Quanyi Li et al. describe the same trade-off for reward penalties), so early violations should be higher. If Safe-CE² equals our system on EMCC in the irreversible variants, the tether is not worth its complexity, and we will report that.

Why not just use SafeDreamer with more samples? Monolithic exploration covers distance roughly like a random walk, as the file notes. Sparse-goal mazes then need many more steps, and the KEA and TLDR results in the file show monolithic curiosity methods stuck in initial corridors. The steps-to-success curves at matched cost will show whether SafeDreamer's safety comes at a large discovery cost.

Why not just use SOOPER's fallback or a formal shield? SOOPER needs a prior in hand. Once we supply the go-phase policy as that prior, the remaining difference is the return-distance check, which the SOOPER-with-learned-prior baseline isolates directly. Adaptive probabilistic shielding builds interval-MDP shields over state spaces of at most a few thousand discrete states and does not extend to continuous MuJoCo control.

Why not just let FDPI's dual policy supply boundary data? It works in simulation but generates real violations. Our replay-buffer and calibration diagnostics test whether near misses give comparable safety-model accuracy without violations.

Proposed Experiments

The environments are the Point-Maze and Ant-Maze tasks used by CE² and the AntMaze-Large tasks used by TLDR, each augmented with cost regions placed along corridor edges. We also build variants with one-way drops, which enter a costly region from which return is impossible. A third set is sparse-reward versions of Safety-Gymnasium PointGoal and CarGoal. Metrics are success rate, state coverage, cumulative training cost, the fraction of episodes with any violation, and EMCC from Revisiting Safe Exploration. EMCC matters because cumulative cost cannot tell a brief touch of a hazard from getting stuck inside one.

The decisive checks are three. First, on the one-way variants, the symmetric-distance ablation should show markedly more unrecoverable violations than the full method; if it does not, asymmetry is not the operative ingredient. Second, coverage per step should stay near unconstrained CE² while cost approaches SafeDreamer. Third, calibration of $C_\phi$ should not degrade late in training, the failure FDPI documented, despite near-zero real violations.