Safety Read Off Successor Features: Imitation That Never Does More Harm Than the Expert

An agent that learns by imitating demonstrations has to practice on its own. Only then does it see the states it drifts into when its actions differ slightly from the expert's, and learn to recover from them. That practice is exactly when robots crash. Current fixes each need something expensive: a supervisor who labels every drifted state, a reachability solver that does not scale, or a large offline dataset that already contains unsafe examples. We propose a cheaper source of safety that is already present in a standard imitation setup.

Suppose the reward is a weighted sum of known features of the state and action, such as "distance to goal" or "proximity to a hazard." Then any policy can be summarized by how much of each feature it accumulates over time. If the agent is told only which features are harmful, and not how harmful, one condition is enough to guarantee it is at least as safe as the expert under every cost built from those features: the agent must accumulate no more of each harmful feature than the expert did. We turn this condition into a constraint for online imitation, and into a run-time guard that blocks exploratory actions predicted to break it. Both are computed from the same learned quantity. The agent can then explore off the expert's path and learn recovery behavior without cost labels, without a supervisor, and without a dynamics model.

Core Claim

The linear reward assumption says the reward is $R(s,a)=w^T\phi(s,a)$, where $\phi$ is a known feature vector and $w$ is unknown. The expected discounted return of a policy $\pi$ is then $w^T\mu(\pi)$. Here $\mu(\pi)$ is the policy's feature expectation: the discounted sum of features it accumulates. The state-conditioned version of $\mu$ is the successor feature $\psi^\pi(s,a)$, the expected discounted future features from $(s,a)$ (Successor Features for Transfer in Reinforcement Learning.).

Our assumption is modest. We assume some feature indices $H$ are hazard features, such as proximity to obstacles or speed above a limit. We also assume the true cost is some unknown non-negative combination of them, $c=w_c^T\phi$ with $w_c\ge 0$ supported on $H$. The expert is taken to be acceptably safe. Under these assumptions, the dominance constraint

$$\mu_k(\pi)\le\mu_k(\pi_E)+\delta\quad\text{for all }k\in H \;\;\Longrightarrow\;\; w_c^T\mu(\pi)\le w_c^T\mu(\pi_E)+\delta\lVert w_c\rVert_1\ \text{ for every admissible } w_c$$

guarantees expert-level safety up to a slack $\delta$ for every cost the designer might have meant. The implication follows from non-negativity of $w_c$. The same componentwise-dominance argument is the one A Game-Theoretic Approach to Apprenticeship Learning uses for rewards; we apply it to costs.

The constraint is $|H|$ ordinary constraints of a constrained Markov decision process (a CMDP: maximize return subject to expected cumulative cost staying under a budget). Each constraint's value function is simply the $k$-th coordinate of $\psi^\pi$. We therefore propose Dominance-Constrained Successor Feature Matching (DC-SFM), which has two parts:

  1. The first part is non-adversarial successor feature matching (Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching) on the non-hazard coordinates. The hazard coordinates are subject to the dominance inequality through a Lagrangian with one multiplier per hazard feature.
  2. The second part is a successor-feature shield: a run-time check using a pessimistic ensemble of $\psi$ networks. It lets an exploratory action through only if the predicted hazard accumulation stays within the budget still unspent in the episode. That prediction covers the action followed by a fallback policy.

The claim is that this combination learns closed-loop recovery behavior with far fewer training-time hazard excursions than safety-agnostic online imitation, and with no cost labels.

Why These Pieces, and What Each Prior Work Contributes

Successor feature matching supplies the method. It already corrects drift the way online inverse RL does, by rolling out the learner and matching its successor features to the expert's. It is stable and works from a single demonstration. It is also explicitly safety-agnostic, and its authors note that it does not solve exploration.

Safety-Constrained Policy Transfer with Successor Features (SFT-COP) shows that successor features can carry both reward and cost values in a constrained Lagrangian setting. It assumes known reward and cost weights, discrete actions, and transfer rather than imitation. We keep its decoupling but drop the known cost weights and move to continuous imitation.

Two papers motivate the problem. SAFE-GIL documents the "expert blind spot": demonstrations keep wide safety margins and so contain no recovery data. It fills the gap with Hamilton-Jacobi reachability and an active expert. OSIL infers a cost model from offline non-preferred trajectories and diverges when coverage is thin. Our route needs neither a reachability oracle nor unsafe examples, only the sign of a few features.

Three papers constrain the design:

  • Safe Exploration via Policy Priors (SOOPER) shows that pessimistic cost prediction plus a fallback prior can keep violations at zero during learning. It also shows that such guarantees are only as good as the prior and the uncertainty calibration. We inherit that weakness and test it directly.
  • Breaking Safety Paradox with Feasible Dual Policy Iteration warns that safer policies see fewer violations, so their cost critics degrade. Hazard features are graded proximities rather than rare violation events, so the constraint critics receive signal long before any violation occurs. We expect this to blunt the paradox, and we test that expectation.
  • Safety-Aware Apprenticeship Learning (CEGAL) pushes feature-expectation learners away from counterexample paths, but needs exact model checking over known discrete dynamics. DC-SFM is model-free.

Research Program

We will build DC-SFM on the SFM actor-critic.

Hazard features. The feature map $\phi$ includes Safety Gymnasium's hazard-lidar channels and a velocity-excess term, which together form $H$, plus task features for the matched coordinates. The expert's hazard occupancy $\mu_H(\pi_E)$ is estimated directly from demonstrations.

Constraints. Each hazard constraint has a multiplier. The multiplier is updated by dual ascent on the gap between the learner's estimated $\psi_k$ at the initial state and $\mu_k(\pi_E)+\delta$.

Shield and fallback. The fallback policy is behavioral cloning (BC) on the demonstrations. We learn the fallback's successor features from demonstration and learner data, using inverse temporal-difference learning as in PsiPhi-Learning. At each step the shield compares two quantities: the accumulated discounted hazard so far, plus the ensemble's upper bound on hazard successor features under "take the proposed action, then follow the fallback." If this sum would exceed the expert budget on any coordinate, the fallback action is executed instead. Following SOOPER, shielded transitions terminate imagined rollouts in the critic's targets, so the learner is pushed to stop relying on the fallback.

Analysis. On the theory side, we will turn the implication above into a bound on the learner's true cost. The bound is in terms of successor-feature estimation error on the hazard coordinates, measured in the $\ell_\infty$ norm (the largest error on any single coordinate) rather than the $\ell_2$ norm SFM minimizes. We will also characterize when the shield is safe, namely when the ensemble upper bound covers the true fallback hazard successor features.

Baselines

Behavioral cloning (naive reference). Plain BC on the same demonstrations is the naive reference point. Datasets and Benchmarks for Offline Safe Reinforcement Learning reports that BC on all data earns reward but violates constraints, while BC on safe data only is overly conservative. SAFE-GIL reports roughly 80% failure for BC in ground navigation with 8 demonstrations. Beating BC on both task success and hazard occupancy is necessary but not interesting on its own.

Unconstrained SFM (strongest imitation competitor). This is the strongest competitor because it already corrects drift through online rollouts. Exact matching would already give expert-level safety for every linear cost. It reports up to 16% higher mean normalized return than competing methods on DMC. We found no reported result on Lacuna for SFM on Safety Gymnasium, so we will run it ourselves with the same features, demonstrations, network sizes, environment-step budget, and hyperparameter search budget as DC-SFM. DC-SFM beats it if it cuts cumulative training-time hazard excursions and final cost by a clear margin across seeds, while keeping normalized return within noise.

SFM plus PPO-Lagrangian with true cost labels (oracle upper bound). This baseline shows what is lost by not knowing $w_c$. Safety Gymnasium reports that PPO-Lagrangian achieved a 98% reduction in violations at a 45% reward drop relative to PPO on velocity tasks (Safety Gymnasium). We do not expect to beat this oracle. Matching it at convergence without labels would count as success.

OSIL (strongest label-free safe-imitation method). OSIL is given extra non-preferred trajectories from DSRL; it reports nearly 2.8x the best baseline in one setting. We run it on the same DSRL tasks. DC-SFM beats it if it achieves equal or lower cost without any non-preferred data, particularly in the low-coverage regime where OSIL's authors expect divergence.

Ablations. Two ablations isolate the key ingredients. The first is DC-SFM without the shield, which isolates safety during learning from safety at convergence. The second is DC-SFM with the dominance constraint replaced by a single scalar hazard penalty of fixed weight, which isolates the multi-constraint structure.

Why not just…?

Why not just run SFM and trust that matching feature expectations already implies matching safety? This is the real threat. At exact convergence it is correct. However, SFM minimizes an unweighted $\ell_2$ gap. A small overall error can be concentrated in the hazard coordinates, and during training the gap is large. We will measure SFM's hazard-coordinate excess over training and at convergence. If that excess is negligible and training-time excursions are comparable to DC-SFM's, the dominance constraint is unnecessary and we will report that the shield is the only useful contribution.

Why not just subtract a hand-tuned penalty on hazard proximity? The corpus reports that fixed penalties either tolerate failures or collapse into policies that do nothing useful (Quanyi Li et al.), and that SFT-COP's risk-penalized baseline became overly conservative. The scalar-penalty ablation, swept over penalty weights with the same tuning budget, tests whether a single well-chosen weight matches the per-feature dominance constraint.

Why not just use ThriftyDAgger? ThriftyDAgger reports a 73% autonomous success rate on simulated peg insertion, but it needs a human to take over in risky states. Our setting assumes no supervisor after the demonstrations. We will report ThriftyDAgger with a scripted expert as an upper reference and count its expert queries. The claim is not higher success than ThriftyDAgger, but comparable safety with zero queries.

Why not just learn a cost model from unsafe trajectories, as OSIL does? OSIL works when unsafe examples exist and cover the relevant states. The sign-only assumption is weaker and needs no unsafe data. The head-to-head comparison on DSRL, with OSIL's non-preferred data subsampled, will show where each assumption pays off.

Proposed Experiments

The main testbed is Safety Gymnasium navigation (Point-Goal1, Car-Goal1, Point-Button1) and velocity tasks (Swimmer- and Ant-Velocity). We use 1, 5, and 25 expert demonstrations taken from the safe trajectories in DSRL. The metrics are:

  • normalized return;
  • final episode cost under the benchmark's true cost;
  • cumulative cost incurred over all training rollouts, which is the safety-during-learning measure;
  • shield activation rate over training.

All results are reported over at least five seeds.

Four checks are decisive:

  1. DC-SFM's cumulative training cost is well below unconstrained SFM's at matched return.
  2. Final cost approaches the labeled oracle.
  3. Robustness to misspecification. We replace the benchmark cost with a threshold-on-contact cost that is not in the non-negative span of $H$, and measure how much the guarantee degrades.
  4. Stress tests of the shield's dependencies. We shrink the ensemble to degrade calibration, and weaken the fallback by training BC on a single demonstration. We then measure how quickly training-time violations grow.

A clean positive result on the first three, with graceful degradation on the fourth, would establish that sign knowledge about features is enough to make online imitation safe while it learns to recover.