Many safe reinforcement learning tasks can be solved by several qualitatively different routes: pass an obstacle on the left or on the right, or reach under or over a joint limit. Routes that cannot be bent into one another without crossing a forbidden region belong to different homotopy classes. Standard constrained policy-gradient methods improve a policy by deforming its behavior continuously. A policy that has settled on the worse route therefore faces a bad choice. It can drive its behavior straight through the forbidden region to reach the better route, which violates safety. Or it can stay trapped. Transfer Reinforcement Learning across Homotopy Classes documents this trap. It escapes only by temporarily switching the safety penalty off, and it needs the barrier's geometry to be known in advance.
We propose a way out that never switches safety off. The idea is borrowed in spirit from condensed-matter physics. There, strongly interacting systems can change their topological class while the energy gap protecting them stays open. The change happens through collective modes rather than through a dangerous single-particle excitation. In our setting the protected quantity is the safety margin: the minimum distance, or cost-critic slack, between visited states and the unsafe set. The "collective" route is to change the probability mass the policy places on each route, rather than to bend a single route through the obstacle.
Concretely, the method has three steps. It first discovers a safe trajectory in a new homotopy class using a shielded, class-novelty-seeking explorer. It then adds that trajectory as a new component of a mixture policy. Finally, it transfers weight to the new component continuously. Along the whole way the visitation probability of the barrier stays exactly zero. The quantity that jumps is only a coarse summary, the route taken by the policy's average trajectory.
Core Claim
Consider a behavior family that moves from class $H_1$ to class $H_2$ by continuous deformation of a single unimodal route. Such a family must at some point assign positive visitation to the barrier set $S_b$. Now consider moving along mixture weights between two individually safe components. That path keeps the barrier visitation at zero and keeps the minimum safety margin bounded below by the smaller of the two components' margins.
Write $d_\pi(S_b)$ for the discounted state-visitation probability of the barrier under policy $\pi$. For a mixture $\pi_w = (1-w)\pi_1 + w\pi_2$, chosen at episode start, the visitation is linear in $w$:
$$d_{\pi_w}(S_b) = (1-w)\,d_{\pi_1}(S_b) + w\,d_{\pi_2}(S_b) = 0 \quad \text{for all } w \in [0,1] \text{ when } d_{\pi_1}(S_b)=d_{\pi_2}(S_b)=0.$$
The same linearity holds for the cumulative-cost constraint of a constrained Markov decision process, which requires expected discounted cost to stay below a budget $\epsilon$. Weight transfer between two feasible components is therefore feasible at every step.
The mathematics here is elementary. The claim is empirical and algorithmic. The hard part is finding a safe policy in $H_2$ without ever entering $S_b$ and without knowing where $S_b$ is. That discovery step is what we build and test.
The physics serves as a lens and a diagnostic, not as a method we import. In Topological number and fermion Green's function for strongly interacting topological superconductors and Exotic quantum phase transitions of strongly interacting topological insulators, the topological number computed from the single-particle propagator changes at a zero of that propagator rather than a pole. The single-particle description loses meaning while the physical gap stays open. Bona fide interaction-driven topological phase transition in correlated symmetry-protected topological states shows by quantum Monte Carlo that such transitions are driven entirely by collective bosonic fluctuations.
Our analogue is as follows. The winding number of the mean trajectory is the "single-particle" summary. It jumps when the mean path passes through the obstacle, yet no sampled path does. The barrier visitation $d_\pi(S_b)$ plays the role of the gap-closing signal, and it stays at zero. The analogy gives us a sharp, measurable prediction: a mean-route invariant that jumps while barrier occupancy stays flat. Unimodal methods should instead show a spike in barrier occupancy at the moment of transfer.
Research Program
The system has three parts.
Barrier estimation and route labeling. A cost critic is trained on observed costs, in the style of the peak-cost feasibility score of Feasibility Consistent Representation Learning for Safe Reinforcement Learning. That paper smooths the safe/unsafe boundary into a scalar level set. We reuse its score, but for a different job: we extract an estimated unsafe region with a safety margin, then compute a route label for each trajectory relative to that region. The label is a winding number in planar tasks, or a homology-style signature computed around obstacle components in higher dimensions. This removes the need for a priori barrier coordinates that the Ease-In-Ease-Out method of Cao et al. requires.
Shielded class-novelty exploration. An explorer policy is rewarded for producing trajectories whose route label differs from those already archived. Its actions pass through a state-wise safety filter built from backward value functions. Constrained Markov Decision Processes via Backward Value Functions turns a trajectory-level cost budget into a local test at each state: cost so far plus expected future cost must stay within budget. We use that test to reject actions, not to shape the explorer's updates.
We keep this shield fixed. Its known conservatism may block narrow routes, and we measure how often that happens.
This is the opposite design choice to Breaking Safety Paradox with Feasible Dual Policy Iteration. That method keeps its feasibility estimate sharp by letting a second, reckless policy deliberately violate constraints. We accept a blunter boundary estimate in exchange for zero intended violations. The estimate is then sharpened only through near-boundary safe samples that the novelty bonus naturally produces.
Mixture weight transfer. Each discovered class seeds a policy component. Each component is refined inside its own class with a boundary-tracking constrained optimizer, following Boundary-Seeking Policy Gradient for Safe Reinforcement Learning. That method moves the policy along the cost boundary to use the full budget, but only within the current feasible region. We use it inside each class and never across classes.
A gating weight over components is then updated by return comparison. The update is subject to the linear constraint above, so every intermediate policy is feasible whenever every component is.
Analysis. Beyond the experiments, we will write out two results. The first is the zero-occupancy argument for mixtures. The second is a matching lower bound: under a continuous state-feedback policy with a unimodal action distribution, any continuous path of routes between classes must visit $S_b$. The standard fact that homotopic paths must intersect the obstacle gives this bound. We will also state precisely where it fails, namely for stochastic policies with heavy tails.
Baselines
The naive reference is PPO-Lagrangian, i.e. constrained proximal policy optimization with an adaptive multiplier, as implemented in the SafePO library of Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark. It is initialized in the worse class. That paper reports that PPO-Lag achieves a 98% reduction in safety violations relative to PPO with a 45% drop in reward on velocity tasks. We expect it to stay in $H_1$ with low violations. "Beating" it means reaching $H_2$'s higher return.
The strongest method for actually changing class is Ease-In-Ease-Out fine-tuning, reported in Transfer Reinforcement Learning across Homotopy Classes to reach target reward on Lunar Lander in about 80k steps, where naive fine-tuning failed to converge after 300k steps. We run it with the true barrier geometry it requires, same environment steps and same tuning budget. The comparison target is to match its transfer success and step count within a factor we report. At the same time we must incur far fewer training-time violations: EIEO's are nonzero by design, and ours should be zero or near it. If we lose badly on steps, we report that trade-off honestly rather than claim dominance.
The strongest pure safe-RL competitors are FOCOPS, from First Order Constrained Optimization in Policy Space, and BSPG. FOCOPS reports 1106.1 reward on Humanoid-Circle versus 671.0 for CPO under a cost limit of 50. For BSPG on SafetyPointGoal1-v0, no reported scalar result was found on Lacuna. FDPI is included as a violation-tolerant upper reference for return. The file gives no scalar for it either, only "near-zero" final violations across 14 environments.
All methods share the environments, network sizes, number of environment steps, and a fixed hyperparameter sweep budget per method.
Three ablations isolate the key ingredients. The first uses the same mixture architecture but lets components be initialized randomly and trained with a Lagrangian objective, with no novelty explorer. The second keeps the explorer but replaces weight transfer with distilling a single Gaussian policy onto the $H_2$ trajectory. The third keeps the explorer but hard-switches to the best component with no gradual transfer.
Why not just…?
Why not just relax the penalty temporarily, as Ease-In-Ease-Out does? It works in simulation, and it is our main comparison. But it deliberately drives through the barrier during the relaxation phase, and it needs barrier coordinates. Our experiments count violations during transfer. If EIEO's count is not meaningfully larger than ours on the homotopy benchmarks, the safety motivation for our method collapses and we will say so.
Why not just use a multimodal policy, such as a Gaussian mixture, with a Lagrangian objective? This is the most serious threat. The gradual-transfer argument is available to any mixture policy. Our bet is that without targeted discovery, no component ever lands in $H_2$. Components would either collapse onto $H_1$, or entropy would push mass toward the barrier and the constraint would push it back. The first ablation tests exactly this. If it matches the full method, the contribution shrinks to "use mixtures," and we would retarget the work to the discovery problem alone.
Why not just train a separate safe policy in the other class and switch to it? Once a safe $H_2$ trajectory is found, a hard switch is also safe. The mixture's continuous transfer mainly buys on-policy evaluation of the switch under the budget. We expect the explorer, not the transfer rule, to carry most of the gain. The hard-switch ablation measures this directly, and we treat the discovery step as the core contribution.
Why not probe the boundary with a dual reckless policy, as FDPI does? FDPI's own limitation is that the dual policy violates constraints in the real environment. That rules it out for the hardware-style setting we target. We include it only to measure how much return we give up by refusing to cross.
Proposed Experiments
We use the five homotopy benchmarks of Cao et al.: 2D navigation, Lunar Lander, Fetch Reach, MuJoCo Ant, and Assistive Feeding. Their barriers are recast as cost constraints with budget $\epsilon = 0$ for barrier entry. We add Safety-Gymnasium variants of SafetyPointGoal1-v0 and CarGoal, in which hazard layouts are fixed so that two routes to the goal exist with different lengths, and agents start from a policy in the longer route.
Metrics are final return; cumulative training-time violations; minimum safety margin over all training episodes; fraction of seeds (at least ten per condition) that end in the better class; and environment steps to transfer.
The decisive checks are three. First, the full method must reach $H_2$ in a majority of seeds with zero barrier visits, while PPO-Lag, FOCOPS, and BSPG remain in $H_1$. Second, the gap-preservation diagnostic must show the predicted signature. Logged over training, the mean-trajectory winding number should jump while barrier occupancy and minimum margin stay flat for our method, and occupancy should spike for EIEO and for the Gaussian-distillation ablation. Third, the mixture-with-Lagrangian ablation must fail to discover $H_2$. If it succeeds, the project's claim reduces to the discovery module, and we report it that way.
We also report how often the backward-value shield blocks a narrow but safe route. This quantifies the conservatism cost that BVF itself leaves open.
References
- Transfer Reinforcement Learning Across Homotopy Classes
- Topological number and fermion Green's function for strongly interacting topological superconductors
- Exotic quantum phase transitions of strongly interacting topological insulators
- Bona fide interaction-driven topological phase transition in correlated symmetry-protected topological states
- Feasibility Consistent Representation Learning for Safe Reinforcement Learning
- Constrained Markov Decision Processes via Backward Value Functions
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Boundary-Seeking Policy Gradient for Safe Reinforcement Learning
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- First Order Constrained Optimization in Policy Space