Robots that physically assist people, such as powered leg braces (exoskeletons) that help a wearer walk or carry loads, are often trained by imitation. The robot copies recorded examples of good assistance. Imitation has a well-known weakness. Once the robot makes a small mistake, it drifts into situations its examples never covered, and the mistakes pile up over time. The standard remedy is to have an expert correct the robot live in exactly those drifted situations. That remedy is unavailable here: a person strapped into a device cannot also teleoperate it or dictate joint torques. There is also a second, less discussed problem. The robot's training environment is usually a human "played back" from motion capture, but a real wearer reacts to the robot's mistakes by bracing, pushing, or changing stride. The drift the robot meets at deployment is therefore partly created by the human it is supposed to help.
We propose to turn that reaction into the correction signal. When an exoskeleton assists wrongly, the wearer pushes against the straps (cuffs) that connect them to it, and the direction and size of that push say which way the assistance should have gone. The plan is to deliberately and safely nudge the policy off its demonstrated behavior, staying within a strict budget on cuff force. At the resulting drifted states, the wearer's compensating force is read as a corrective action label. The policy is then retrained on those labels, much as an interactive imitation method aggregates expert corrections. Put in two sentences: the coupled human stands in for the missing teacher, and safe exploration decides where to ask.
Core Claim
In a physically coupled human-robot pair, the interaction force measured at the contact points contains a usable corrective signal at off-distribution states. Collecting that signal through force-budgeted perturbations shrinks the compounding error of imitation-learned assistance to a degree that neither passive cloning nor force-minimizing reinforcement learning against a played-back human can match. The mechanism is new for three reasons. The label source is the coupled human, not an external supervisor. Exploration is constrained by a biomechanical cost, cuff force, rather than collision. And training uses a reactive human model, not motion playback, so the distribution shift being corrected is the one caused by human adaptation.
Concretely, let $a_t$ be the exoskeleton's commanded assistance, for example joint torque or virtual impedance parameters, and $f_t$ the vector of cuff interaction forces at the back, thighs, and shanks. The force-derived label is
$$\hat a_t = a_t - G\,(f_t - \bar f_t),$$
where $\bar f_t$ is the cuff force expected under correct assistance in that gait phase, estimated from demonstrations, and $G$ is a gain matrix that maps force residuals to action corrections. $G$ is fit from demonstration data with small known perturbations. Exploration injects a perturbation $\delta_t$ into $a_t$ only when the force budget allows it:
$$C_{\text{past}}(x_t) + \hat C_{\text{future}}(x_t, \delta_t) \le B,$$
where $C_{\text{past}}$ is the accumulated excess cuff force so far in the episode, $\hat C_{\text{future}}$ is a learned estimate of the excess force still to come, and $B$ is a per-episode budget.
Relation to Prior Work
The problem is motivated by the exoskeleton control literature. Zheng et al. train a reinforcement learning agent to tune virtual mass, damping, and gravity parameters that reduce interaction force (Virtual Impedance Adaptation of Lower-Limb Exoskeleton for Human Performance Augmentation Based on Deep Reinforcement Learning). A related end-to-end controller (End-to-End High-Level Control of Lower-Limb Exoskeleton for Human Performance Augmentation Based on Deep Reinforcement Learning) takes the same general approach. Both drive the human by prerecorded kinematics, so the simulated human never reacts to a robot error. We keep their spring-damper cuff contact model but replace the playback human with a feedback-controlled one. Human-Robot Gym has the same playback limitation. Its safety shield brakes to a stop whenever contact is predicted, and its authors explicitly call for shields that permit intended contact. Our force-budget gate is one answer to that call.
The label idea descends from interactive imitation. DAgger (Mitigating Compounding Errors in Behavioral Cloning) queries an expert at the robot's own visited states and reduces error growth from quadratic to linear in horizon. ThriftyDAgger rations those queries with novelty and risk gates plus hysteresis to spare the supervisor. SAFARI predicts out-of-distribution states with a world model and asks for demonstrations there. All three assume a teleoperating supervisor who is outside the physical loop. We borrow ThriftyDAgger's gating and hysteresis, and SAFARI's idea of choosing where to query, but we substitute a label read from the coupled human.
Two further lines constrain the design. Closing the Closed-Loop Distribution Shift in Safe Imitation Learning shows that imitation sample complexity depends on how strongly the closed loop pulls trajectories back together. The authors call this incremental gain stability; a "contracting" loop is one where nearby trajectories converge. Their stability-constrained methods require known dynamics. We use their analysis diagnostically, estimating the contraction of the coupled loop from data to predict when plain cloning will fail. We do not rely on their constraint. The backward value functions of Constrained Markov Decision Processes via Backward Value Functions supply the budget inequality above, and they keep safety intact during training rather than only at convergence.
Finally, Safe and Efficient Exploration of Human Models During Human-Robot Interaction finds that learning how a human reacts to the robot requires risk-seeking probes. Its risk-seeking setting triggered 19.7 safety-monitor interventions per 100 timesteps. On a worn device that rate would feel like constant jerking. This is a direct warning against boundary-hugging exploration, and it is why our perturbations are budgeted per episode and smoothed with hysteresis rather than projected step by step.
Research Program
Reactive co-simulation. The first component is a reactive co-simulation in MuJoCo. It reproduces the lower-limb exoskeleton setup of Zheng et al.: viscoelastic cuffs at back, thighs, and shanks, and five walking speeds from 2.8 to 5.6 km/h. The human, however, is a torque-driven body with its own feedback controller that tracks a reference gait while trading off effort. Compensation delay, stiffness, and gain vary across a population of simulated "subjects," with held-out subjects kept for testing, in the spirit of GlucoSim's held-out patients (Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed).
We deliberately do not use the 700-muscle model of Self Model for Embodied Intelligence. Its gait artifacts and two-stage training make it too slow for a control loop. Instead, it serves as a slower, higher-fidelity stress test for a subset of runs.
Demonstrations and label model. Demonstrations come from a strong model-based assistance controller running with privileged knowledge of each training subject. The force-to-correction gain $G$ is fit from those demonstrations under small known action offsets.
Learning loop. The learning loop alternates two steps. First, it runs the current policy with gated perturbations, where the gate uses ensemble disagreement for novelty and the backward-value budget for safety. Second, it aggregates the force-derived labels and retrains the policy.
Co-manipulation variant. A second testbed modifies the Lifting and handover tasks of Human-Robot Gym so that the human arm is reactive and coupled through the carried load. The question is whether the mechanism is specific to gait.
Where the label breaks. We analyze when the force label is misleading. Wearers adapt their own gait to the device, and a human who "gives up" and rides the device produces small forces despite poor assistance. The simulated population includes such co-adapting subjects precisely to measure this failure.
Baselines
The naive reference point is plain behavioral cloning on the privileged-controller demonstrations. It is trained on the same data and evaluated on the reactive human. Its failure mode is the one the proposal targets, so it sets the floor. A second reference, which isolates the playback-versus-reactive question, trains the same cloned policy in playback co-simulation and then evaluates it reactively.
The strongest existing method for this exact problem is the reinforcement-learning virtual-impedance controller of Zheng et al. (Virtual Impedance Adaptation of Lower-Limb Exoskeleton for Human Performance Augmentation Based on Deep Reinforcement Learning). It is the published approach that directly optimizes interaction force on this device class. It reports a lumped interaction-force ratio of 0.81 against a sensitivity-amplification baseline in its Simscape simulation, and 0.89 on the physical prototype. Those numbers come from a different simulator with a playback human, so they are context, not a target. We found no reported result on Lacuna for any method on a reactive-human exoskeleton benchmark. We will reimplement the method with its three-parameter action space and TD3, give it the same simulator-step budget as our full pipeline including demonstration generation, and tune it with the same number of hyperparameter trials. We also run it trained against the reactive human, so that it is not handicapped by the playback assumption. Beating it means a lower lumped force ratio on held-out subjects at all five speeds with no more cuff-budget violations during training.
The strongest safe-learning competitor is PPO-Lagrangian with cuff force as the constrained cost, taken from the SafePO library in Safety Gymnasium. It gets the same budget $B$, compute, and tuning trials. We report its training-time violations as well as its final behavior.
The ablations isolate the key ingredient. The first uses force labels collected without perturbation, which shows how much the active exploration adds. The second uses unbudgeted random noise injection in the style of DART, with force labels, which shows how much the safety gate costs or saves. The third uses budgeted perturbation with oracle labels from the privileged controller. It gives the ceiling for how good the force label could be, and the gap between it and the full method measures the label's error.
Why not just…?
Why not just use reinforcement learning to minimize interaction force directly? This is the most serious alternative. With a reactive human in the simulator, the reward already carries the wearer's reaction. The difference is sample efficiency and safety during learning. Unconstrained model-free learning on rich human models can collapse entirely: Self Model for Embodied Intelligence reports a return of 14.72 for end-to-end reinforcement learning versus 371.07 with structured training. Force-labeled imitation instead starts from good demonstrations and needs only local corrections. The comparison against the reactive-trained Zheng et al. controller at equal simulator steps settles the question. If that controller matches our method in held-out force ratio and training violations, the label idea adds nothing and we will say so.
Why not just add a penalty for large cuff forces? Safety Gymnasium shows that scalar penalties either get traded away for reward or freeze the policy. The diabetes testbed shows that penalty-shaped and Lagrangian policies lose over 10% of time-in-range on unseen patients, a safety-generalization failure. The PPO-Lagrangian baseline, reported with training-time violations over held-out subjects, tests whether the same holds for cuff force.
Why not just clone more demonstrations, or use stability-constrained cloning? More demonstrations help only if the coupled loop contracts. The stability-constrained methods of Tu et al. need known dynamics, which soft cuffs and an adapting wearer do not provide. We estimate the coupled loop's contraction for each subject and check whether it predicts where plain cloning fails, as their theory says it should. If some subjects turn out to contract, cloning with extra data should suffice for them, and our gains should concentrate on the non-contracting ones.
Proposed Experiments
Label informativeness. The first decisive check is whether the force label carries information at all. On held-out subjects, we measure the angle and magnitude error between $\hat a_t$ and the privileged-controller action at perturbed states. We report it separately for compliant, stiff, delayed, and co-adapting subjects. If the error is not clearly below that of zero-correction labels for most subject types, the premise fails and the project reduces to the co-simulation benchmark.
Deployment shift. The second check measures shift directly. We plot tracking error and lumped force ratio against episode length for cloning trained on playback versus reactive humans. This tests whether the error growth predicted in Mitigating Compounding Errors in Behavioral Cloning appears only with a reactive human, and whether force-labeled aggregation flattens it toward the linear growth DAgger achieves.
Main comparison. The main comparison reports the following against every baseline above:
- held-out lumped force ratio at each walking speed;
- falls or gait-tracking failures;
- cumulative cuff-budget violations during training;
- the number of perturbation episodes needed to reach the oracle-label ceiling.
The comparison is repeated on the Human-Robot Gym lifting and handover variants with a reactive arm.
Robustness to model fidelity. A final stress test runs a subset of trained policies against the 700-muscle human, to check whether gains survive a change in the underlying human model rather than being an artifact of our reduced one.
References
- Virtual Impedance Adaptation of Lower-Limb Exoskeleton for Human Performance Augmentation Based on Deep Reinforcement Learning
- End-to-End High-Level Control of Lower-Limb Exoskeleton for Human Performance Augmentation Based on Deep Reinforcement Learning
- Human-Robot Gym: Benchmarking Reinforcement Learning in Human-Robot Collaboration
- Learning monocular reactive UAV control in cluttered natural environments
- ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
- SAFARI: Safe and Active Robot Imitation Learning with Imagination
- Closing the Closed-Loop Distribution Shift in Safe Imitation Learning
- Constrained Markov Decision Processes via Backward Value Functions
- Safe and Efficient Exploration of Human Models During Human-Robot Interaction
- Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed
- Self Model for Embodied Intelligence: Modeling Full-Body Human Musculoskeletal System and Locomotion Control with Hierarchical Low-Dimensional Representation
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Human-compatible driving partners through data-regularized self-play reinforcement learning