A robot that learns by copying demonstrations drifts. Each small error moves it into states the demonstrator never visited. Interactive methods such as DAgger fix this by letting the learner act and having an expert take over, and relabel, when things go wrong. The hard part is deciding when to hand control back. Existing systems decide by asking whether the current state looks unfamiliar to the learner's policy. From raw camera images, "unfamiliar" is measured in whatever features the policy happened to learn for the task. Those features have no reason to register a thin obstacle, a nearby hazard, or a slowly building dangerous velocity. They do react strongly to harmless changes in lighting or background. The result is a gate that fires too often on changes that don't matter and too late on changes that do.
We propose to learn a separate, small representation whose only job is to encode how close the robot is to needing rescue, and to gate expert takeover on shift in that representation rather than in the task representation. The training signal requires no hand-written cost function. It comes from the takeover events the interactive loop already produces, plus the failures that occur when a takeover comes too late. The core claim is that, for a fixed expert-intervention budget, gating on safety-representation shift reduces unsafe events during data collection compared with gating on policy uncertainty. It should also save queries by letting the learner continue through task-irrelevant visual changes. A learned safety representation should approach a hand-engineered safety feature (such as distance to the nearest hazard) without anyone writing that feature.
Where this sits
Three lines of work each hold one piece. The interactive-gating methods address distribution shift and safety together, but on engineered or policy-internal features. EnsembleDAgger and DropoutDAgger hand control back when an ensemble, or dropout samples of the policy, disagree about the action. MMD-IL uses a kernel distance between the current state and the demonstration set, with a manually chosen kernel. ThriftyDAgger adds a learned value estimating the chance of reaching the goal and calibrates thresholds to an operator budget.
These papers motivate the proposal: they show that gating works. They also expose the gap. Every one of their signals is either the policy's own uncertainty or a task-success estimate, and none is trained to track safety specifically. Hand-built features show the cost of getting features wrong. The original monocular UAV DAgger system (Ross et al.) flew on four hand-designed vision filters and still side-swiped obstacles that the filters did not capture.
The representation-learning line supplies method but no safety. VINN shows that a self-supervised encoder trained separately from actions generalizes far better than end-to-end features: 40% success on novel doors against 3.3% for the best parametric baseline. VINN has no notion of cost and no way to ask for help. The research file also notes a general weakness of compression-trained encoders: they discard visually small but safety-critical cues.
The safe-RL line supplies the decoupling technique but assumes cost labels. SDQC splits observations into a reward representation and a cost representation. It trains them with contrastive learning that groups states by similar reward or cost values, and it shows this abstraction is more robust to out-of-distribution feature combinations. SDQC works offline and needs a cost label on every transition, which imitation data does not have. We reuse its contrastive recipe but replace the cost value with an intervention-derived target, and we move it into the online interactive loop.
The closest work is the conformal safety-learning codebase (conformal-safety-learning, for Learning Robot Safety from Sparse Human Feedback using Conformal Prediction). It also learns a safety filter from sparse human interventions and hooks into CLIP visual features. From the file we can only tell that it calibrates a filter over given features. Our proposal differs in learning the feature space itself and in separating task shift from safety shift. We treat this work as a baseline, not a solved version of our question.
The formal-safety line constrains the design. SAFE-GIL forces experts to demonstrate recoveries near the safe-set boundary using Hamilton-Jacobi reachability, a grid-based computation of which states can still avoid failure. It needs known low-dimensional dynamics. V-OCBF learns barrier functions from offline data, but its filter assumes control-affine dynamics. We give up formal guarantees in exchange for working from pixels, and we say so plainly. Breaking Safety Paradox with Feasible Dual Policy Iteration warns that as a learner gets safer it produces fewer failure examples, starving any learned safety estimator. This is a direct threat to learning from interventions, and the program below addresses it.
Core Claim
Let $z_c = \phi_c(o)$ be a low-dimensional safety embedding of observation $o$, and $z_r$ the task embedding used by the policy. We train a safety target $h(s)$ by temporal-difference learning, with $\gamma$ the discount factor:
$$h(s_t) = \mathbb{E}\left[\sum_{k\ge 0} \gamma^k\, e_{t+k}\right],$$
where $e_t = 1$ if the expert took over or a failure occurred at step $t$. In words, $h(s)$ is a discounted "rescue-is-coming" score. We shape $\phi_c$ with SDQC-style contrastive learning so that states with similar $h$ lie close together.
The gate hands control to the expert when either of two conditions holds. The first is that the predicted $h$ exceeds a threshold. The second is that $z_c$ lies far from the stored set of states known to be recoverable, measured by nearest-neighbor distance, which is the part that catches safety-relevant novelty the predictor has not seen. Thresholds are set as quantiles under an intervention budget, as in ThriftyDAgger. Shift that appears only in $z_r$ does not trigger takeover; the learner continues and the state is logged for later relabeling.
The claim is falsifiable. At matched budget, this gate should produce fewer unsafe events during collection than policy-uncertainty gates. It should produce fewer wasted queries under task-irrelevant visual change. It should come within a small margin of a gate built on a privileged hand-engineered safety feature.
Research Program
We will build the system on a shared visual backbone, a BYOL encoder as in VINN, with two small heads. The task head feeds a diffusion or regression policy. The safety head is trained from intervention and failure events. In simulation, the expert is a privileged scripted or RL policy that intervenes according to a hidden rule, such as "take over when time-to-collision falls below a threshold." This gives a controllable stand-in for a human, which is the protocol ThriftyDAgger's simulation studies use.
To counter the safety paradox, we will keep every intervention-labeled state in the dataset permanently. We will also add cheap near-boundary data by injecting random pushes during a small fraction of expert demonstrations. This is a dynamics-free, crude version of SAFE-GIL's adversarial perturbations. An ablation will measure whether it matters.
The analysis has three parts. First, we test whether $\phi_c$ recovers safety-relevant geometry by regressing privileged quantities (hazard distance, speed toward the obstacle) from $z_c$ and from $z_r$. Second, we check whether the two embeddings shift independently under controlled perturbations. Third, we track how the gate's recall of pre-failure states evolves as interventions become rare over DAgger rounds, which directly measures the safety paradox.
Baselines
The naive reference is behavioral cloning on the same demonstrations and encoder, with no interaction. It anchors how much any interactive method helps.
Two reference points bracket the space. Full DAgger, with the expert labeling every visited state, is the label-rich upper bound on task learning, but it provides no protection during collection. A privileged-feature gate uses the true hand-engineered distance-to-hazard and speed in place of $z_c$. It measures how close the learned representation comes to manual feature engineering, and matching it within a few points of safety recall is our target.
The strongest competitor is ThriftyDAgger. It already combines a novelty gate with a learned risk value under a budget, and it reports 73% autonomous success on simulated peg insertion, against 24% for SafeDAgger and 48% for LazyDAgger (ThriftyDAgger). We will run it on the identical BYOL backbone, demonstrations, simulated expert, intervention budget, and hyperparameter search budget. The only difference will then be where the gating signal comes from.
Other baselines run under the same controls. EnsembleDAgger is a second uncertainty gate; its reported result is zero failures on inverted pendulum (EnsembleDAgger), on low-dimensional state. The conformal safety filter from the released codebase runs over CLIP features.
Beating ThriftyDAgger means one of two outcomes at equal budget and equal or better final autonomous success. Either unsafe events during collection drop by a clearly significant margin across seeds, or the same safety is reached with measurably fewer queries.
Three ablations isolate the key ingredient. The first uses a single shared embedding, with the safety loss added to the task representation, and tests whether the separation itself matters. The second uses a safety head on generic BYOL features without the contrastive safety shaping. The third removes the injected near-boundary pushes.
Why not just…?
Why not just use ThriftyDAgger's risk value, which already estimates the chance of failure? Its value predicts reaching the goal. It is trained on the task, lives on the policy's features, and treats slow progress and imminent collision identically. It is nonetheless a real threat: if the value alone catches pre-failure states as well as our gate, the separate safety representation is unnecessary. The matched comparison, and especially the safety-only-shift test below, settles this directly.
Why not just use a strong pretrained encoder with ensemble uncertainty? VINN shows that generic self-supervised features generalize well for task imitation. The concern is that compression-trained features underweight small hazards. We test this by regressing hazard distance from generic features versus $z_c$, and by comparing gates. If generic features plus ensembles match us, we will report that the representation claim fails.
Why not just run constrained RL with a cost critic, as in backward value functions or V-OCBF? Backward value functions need an environment cost signal and a feasible starting policy. V-OCBF needs control-affine dynamics for its filter. Imitation data from pixels offers neither, which is exactly the setting we target. Where Safety-Gymnasium does provide costs, we will report a cost-labeled SDQC-style head as an oracle-label variant, to show what the missing labels cost.
Why not just avoid interaction, using action chunking and noise augmentation? Action chunking and data augmentation prove that these can shrink compounding error sharply. They reduce drift but give no mechanism for stopping a dangerous rollout. We include chunked, noise-augmented behavioral cloning as a non-interactive baseline. If it matches interactive methods on both success and unsafe events, interaction itself is not worth its cost in these tasks.
Proposed Experiments
We will use three environments. The first is Safety-Gymnasium navigation (PointGoal and CarGoal with hazards, rendered from pixels), where true costs exist for evaluation only. The second is robosuite/RoboMimic peg insertion and lift, with added contact-force and collision limits. The third is MetaDrive driving from camera input, scored on collision and off-road rates. Each uses 20 to 50 expert demonstrations, five DAgger rounds, a fixed intervention budget, and five seeds.
The metrics are final autonomous success, unsafe events during collection (the central safety-during-exploration measure), expert queries, and recall and precision of the gate on states within $k$ steps of a failure.
Two controlled shifts give the decisive checks. The safety-only shift adds thin or low-contrast hazards along the route without changing the task. Here our gate should fire and policy-uncertainty gates should lag. The task-only shift changes lighting, textures, and distractors while leaving the hazard layout unchanged. Here our gate should stay quiet while uncertainty gates spend budget. If both effects appear and survive the shared-embedding ablation, the claim holds. If ThriftyDAgger's risk value or the privileged-free generic-feature gate matches us on the safety-only shift, the proposal is wrong in its central mechanism, and we will report it that way.
References
- EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning
- DropoutDAgger: A Bayesian Approach to Safe Imitation Learning
- Maximum Mean Discrepancy Imitation Learning
- ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
- Learning monocular reactive UAV control in cluttered natural environments
- The Surprising Effectiveness of Representation Learning for Visual Imitation
- Q-Supervised Contrastive Representation: A State Decoupling Framework for Safe Offline Reinforcement Learning
- Learning Robot Safety From Sparse Human Feedback Using Conformal Prediction
- SAFE-GIL: SAFEty Guided Imitation Learning
- V-OCBF: Learning Safe Filters from Offline Data via Value-Guided Offline Control Barrier Functions
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Constrained Markov Decision Processes via Backward Value Functions
- Action Chunking and Data Augmentation Yield Exponential Improvements in Behavior Cloning for Continuous Spaces