Covariance Tubes: Letting Demonstration Variance Shape Where a Robot May Explore

Robots that learn by trial and error waste most of their attempts, and some of those attempts break things. A common fix is to start from human demonstrations and let reinforcement learning (trial-and-error learning from a reward) correct only what the demonstrations get wrong. Demonstrations carry a second kind of information that is usually thrown away. Where people agree closely, as when inserting a peg, the task is tight. Where they vary, as when reaching across a table, it is loose. A probabilistic movement primitive (ProMP), which is a learned Gaussian distribution over whole trajectories, records exactly this: a mean path plus a time-varying covariance. We propose to turn that covariance into a direction-aware region, a "covariance tube," and to make it the interface between a slow planner and a fast learning controller. Exploration noise is drawn in the tube's shape. A millisecond-scale safety filter keeps the robot inside the tube and outside known obstacles. When the tube and the safe set no longer overlap, the planner is told to re-plan.

The closest prior system, Adapting Object-Centric Probabilistic Movement Primitives with Residual Reinforcement Learning, already showed that demonstration variance helps. It uses the scalar standard deviation of a ProMP as an on/off switch: residual reinforcement learning is enabled only where variance falls below a hand-tuned threshold. On a real Franka arm this converged in 3,000 steps (about 15 minutes), while the ProMP alone succeeded 0% of the time. Our bet is that a scalar switch discards the most useful part of the covariance, which is its direction. A slot is tight laterally but loose along its insertion axis. A cup handle is tight in height but loose in approach angle. A controller that knows which directions are tight can explore much more freely and still stay safe.

Core Claim

Replacing scalar variance gating with an anisotropic, phase-varying tube derived from the ProMP covariance does three things. It lowers the number of environment steps needed to master tight-tolerance tasks. It lowers the number of in-training constraint violations. It removes the hand-tuned threshold. The gain should be largest exactly when task tolerances are anisotropic. Concretely, let $\mu(t)$ and $\Sigma(t)$ be the Cartesian mean and covariance of the ProMP at phase $t$, obtained in closed form from the weight distribution of Probabilistic Movement Primitives. Let $x_t$ be the commanded end-effector position after adding the learned residual. The tube is

$$(x_t - \mu(t))^\top \Sigma(t)^{-1} (x_t - \mu(t)) \le \kappa^2, \qquad h_i(x_t) \ge 0 \;\; \text{for each known obstacle } i,$$

where $\kappa$ is a tube radius in standard-deviation units and the $h_i$ are barrier functions that are non-negative outside obstacle $i$.

We are deliberately not claiming that the demonstration tube is a safety guarantee. It is a learned prior about where the task can be done. The hard guarantee covers only the obstacle barriers $h_i$. For this reason $\kappa$ is not fixed. It is treated as a constraint budget in a constrained Markov decision process: we penalize the time spent near the tube boundary, and a dual variable lets the tube grow when staying inside it is costing reward. This growth matters because the gated method's ProMP failed through tracking lag, so the needed correction may lie partly outside the demonstrated spread.

Research Program

We will build the system in three layers.

The slow layer fits an object-centric ProMP from 5 to 10 demonstrations. At 1 to 5 Hz it conditions that ProMP on the current goal pose and on any via-points, using Bayes' rule, and sends the resulting $(\mu(t), \Sigma(t))$ downward. Unlike the single subgoal images of SuSIE, the latent goal vectors of HiRT, or the deterministic A/RRT waypoints of HiSaRL, what crosses this interface is a distribution. The low level therefore learns which deviations are harmless.

The fast layer is a Soft Actor-Critic residual policy, as in the gated ProMP work. It differs in two ways. Its exploration noise is drawn with covariance proportional to $\Sigma(t)$ instead of isotropically. Its output passes through a projection onto the tube intersected with the barrier half-spaces. With one ellipsoid and a few locally linearized barriers, this projection reduces to a one-dimensional root-find plus a small active-set step. It should run well under a millisecond, in contrast to the offline nonlinear programs over 50 collocation points used by Constrained Dynamic Movement Primitives. We borrow the idea of a fast projection layer beneath a learning policy from ACS, which projects step-wise actions with L-BFGS. Our contribution is the constraint set: ACS's constraint carries no trajectory prior, whereas ours comes from the demonstration covariance.

The feedback channel addresses the "kinematically blind planner" problem documented for SuSIE. When the projection becomes infeasible because a new obstacle cuts through the tube, the fast layer sends the planner the violated barrier. The planner then re-conditions the ProMP on a via-point displaced along that barrier's normal, which is a closed-form operation. This is a narrow, testable kind of feasibility feedback, not a general solution to stale high-level commands.

Orientation will be handled as small rotation-vector residuals around the averaged demonstrated orientation, with a separate diagonal tube. We flag this as a known simplification inherited from the gated ProMP work.

Baselines

The naive reference point is the conditioned ProMP executed alone under the same low-gain impedance controller. The gated work reports 0% success for this on its real insertion task. It tells us how much any learning adds.

The strongest existing method is variance-gated residual reinforcement learning from Adapting Object-Centric Probabilistic Movement Primitives with Residual Reinforcement Learning. It is the only cited system that already uses demonstration variance to limit exploration, so it is the method our claim must beat. Its reported result is convergence in 3,000 real-robot steps (about 15 minutes) on 3 mm-tolerance block insertion. That figure is for hardware. No reported result was found on Lacuna for it in simulation, so we will reimplement it ourselves. For fairness, its threshold will receive the same tuning budget as our tube parameters, a grid of equal size swept with three seeds. Both methods will use identical demonstrations, the same Soft Actor-Critic settings, the same impedance gains, and the same reward. Beating it means reaching 80% success in at least 30% fewer environment steps with no more in-training violations, or reaching equal speed with significantly fewer violations, across five seeds.

The strongest safety-layer competitor is ACS, which reports nearly zero in-training violations and +23.8% cumulative reward over baselines on Kuka-Reach and Kuka-Pick. We will run it on the same residual action space with access to the same demonstrations, so that the comparison isolates the trajectory prior.

We also include a flat constrained learner, PPO-Lagrangian with PID multiplier updates as packaged in Safety-Gymnasium. It will be initialized by behavior cloning on the demonstrations.

Two ablations isolate the key ingredient. The first keeps covariance-shaped noise but removes the projection. The second keeps the projection but replaces the full covariance with its trace, an isotropic tube of the same volume. If the isotropic tube matches the full tube, direction does not matter and the core claim fails.

Why not just…?

Why not just tune the scalar variance threshold more carefully? This is the main threat. On tasks whose tolerances are roughly isotropic, we expect the tuned gate to be about as good as our method. Our experiments therefore include paired tasks that differ only in anisotropy. One is a round peg in a round hole. The other is a flat tab in a long slot, which is tight in one axis and loose in two. If our advantage does not grow on the anisotropic variant, the idea reduces to a smoother gate and we will report it that way.

Why not just use a general safety filter such as ACS with a behavior-cloned initial policy? ACS limits violations step by step. It has no notion of where in the trajectory precision is needed, so its exploration in free space is as broad as anywhere else. The gated ProMP paper showed that this kind of free-space drift is what made always-active residual learning fail. The comparison against ACS with the same demonstrations measures whether the covariance adds sample efficiency on top of a good projection layer.

Why not just scale the exploration noise by the covariance and skip the constraint? That alone may capture much of the sample-efficiency gain. Being cheap, it is our first ablation. Without the projection, however, nothing stops exploration noise from carrying the arm into an obstacle the demonstrations never encountered. We expect that ablation to match our method on steps to success but to lose on in-training violations once obstacles not seen in the demonstrations are introduced.

Why not learn a world model and explore in imagination, as SafeDreamer does? SafeDreamer reports a 94.3% cost reduction over other model-based safe methods, but it trains from scratch per task and is exposed to model exploitation in contact. With 5 to 10 demonstrations and contact-rich insertion, we judge a closed-form trajectory prior the cheaper bet. We will not run it as a primary baseline, and we state that as a limitation.

Proposed Experiments

The primary testbed is a simulated reconstruction of the Ubongo3D-style insertion from the gated ProMP work: a Franka arm under low Cartesian gains, 5 demonstrations, and 3 mm tolerance. It has round and slot variants, and in a third condition a box obstacle is placed across the demonstrated approach. Secondary tasks are Kuka-Reach and Kuka-Pick from the ACS suite, for which we will generate demonstrations by teleoperation or with a scripted controller. Optionally, we will fit ProMPs to bimanual subtasks from DECO-50 to check that real human covariance is anisotropic in the way we assume. No reported baseline numbers were found for that dataset.

We will measure five things: environment steps to reach 80% success, in-training violations (obstacle penetration and contact force above a fixed limit), final success rate, re-plan frequency, and projection latency.

Three checks decide the question. The first is the anisotropy contrast already described. The second is a demonstration-count sweep over 3, 5, and 10 demonstrations with a shrinkage prior on $\Sigma$. Covariances estimated from few demonstrations may be degenerate, and if the method collapses at 3 demonstrations its practical value is narrow. The third is the obstacle-through-the-tube condition. Without the feedback channel the projection should become infeasible and the task fail. With it, success should recover with near-zero penetrations.

If all three checks come out as predicted, the result is a concrete statement. A demonstration's covariance can serve as both the exploration prior and the safety interface in a hierarchical controller, and its directional structure is what earns the gain.