Controllers for exoskeletons, prostheses and rehabilitation robots are almost always tested on able-bodied volunteers who imitate impairment. They wear bypass adapters, use mirrored vision, or have joint limits imposed in software. This matters most for controllers that keep learning on the person, because a learning controller has to try actions it has not tried before. If the healthy stand-in never enters the states where a patient's body behaves dangerously, such as a spastic stretch reflex, a weak muscle giving out, or a fatigued joint drifting to its limit, then an "exploration was safe" result earned on the stand-in says little about the patient. This note proposes to measure that gap in simulation, where the proxy and the patient can be built side by side and harm is free. It then proposes a way to close it: a safety filter that starts from what was learned on the proxy, detects online which parts of a new patient's dynamics differ from it, and becomes cautious only there.
The main claim has two parts. First, standard safe-exploration methods that record near-zero constraint violations while adapting to simulated healthy proxies will record many more violations when adapting to simulated patients built from the same musculoskeletal model. The violations will concentrate in the states where the pathology changes the dynamics. Second, a shield that is widened only in a detected mismatch region can match the violation count of a shield tuned for the worst-case patient while keeping most of the assistance quality that the worst-case shield gives up on milder patients. If the second part fails, the first still stands on its own as a result about how assistive controllers are validated.
What the file supports and what it leaves open
Three groups of prior work set up the problem. The first motivates it by documenting the proxy trap. Natural Multicontact Walking embeds Hill-type muscle models, which describe how muscle force depends on length and velocity, into gait optimization for a transfemoral prosthesis, but validates on two non-disabled subjects using bypass adapters. AR3n trains a Soft Actor-Critic assist-as-needed controller on an abstract kinematic "virtual patient" and tests on eight healthy people tracing with the non-dominant hand under mirrored vision. A Human-Sensitive Controller reports zero pain-zone entries after 5–10 minutes of fine-tuning, but its pain zones are software rules applied to healthy arms. All three handle safety with reward penalties rather than explicit constraints, and none can say what would happen on a spastic or weakened limb.
The second group supplies the method. Provably Safe Sim-to-Real Transfer estimates the region $B$ where reality departs from the simulator. It builds a hybrid model that uses real data inside $B$ and simulation elsewhere, and explores by mixing a new policy with a certified safe one. It proves that real-world samples scale with the size of $B$. That work is tabular and assumes mismatch is either zero or large, which is exactly what continuous biomechanics violates. Sim-to-Lab-to-Real trains a continuous safety-margin critic through a discounted safety Bellman equation, a value function that tracks the worst future distance to failure. It uses that critic as a shield during lab fine-tuning, overriding the policy with a backup whenever the margin runs low. Its setting is quadruped navigation with no human in the loop. We reuse the shield machinery from the second paper and the mismatch-region idea from the first, translated to continuous musculoskeletal dynamics.
The third group constrains the design. Breaking Safety Paradox shows that a safety critic starves once the policy stops producing violations. That is precisely the mechanism by which a proxy-trained critic should be blind to patient-specific danger. Action Projection shows that projecting actions onto a safe set causes gradient collapse or flat value functions, and that its fix costs 3–12x more computation. We therefore use switching to a backup policy rather than differentiable projection. Safety Gymnasium supplies Lagrangian baselines, including PID-damped multipliers, but all of its constraints are geometric.
The simulation substrate is MyoSuite. It runs OpenSim-derived muscle models in MuJoCo 60x to 4,000x faster than OpenSim, with muscle-force error of about 2–4%. It already models fatigue, sarcopenia (age-related muscle loss) and tendon transfer, and it reports that healthy-trained policies fail under sarcopenia. MyoChallenge 2024 adds the myoOSL amputee-plus-prosthesis leg. As far as the file shows, nobody has run safe exploration on these models. Nobody has compared proxy-validated safety against patient safety. And MyoSuite has no spasticity model. Those three absences are where this work sits.
Research Program
The simulated cohort. The first artifact is a paired cohort built on MyoSuite arm models, with an assistive torque actuator added at the elbow and wrist. The task is assist-as-needed trajectory tracing, a musculoskeletal version of AR3n's task. Each "proxy" is the intact model with the impairments that human studies actually impose: extra noise on the motor command, a software joint-range restriction, and added distal mass as a crude stand-in for an adapter. Each "patient" is the intact model with pathology applied to the muscles and reflexes. That means MyoSuite's sarcopenia and fatigue, plus a velocity-dependent stretch reflex that we add ourselves to represent spasticity. The reflex adds excitation to a muscle when its lengthening speed passes a threshold. Individuals are generated by sampling severity, threshold and gain.
Because the spasticity model is ours and not validated against patient data, we implement two different reflex forms: a velocity-threshold form and a length-and-velocity form with a delay. We only claim findings that hold under both. A second, smaller cohort uses myoOSL. There, proxies differ from patients in residual-limb muscle strength and in socket compliance, and falls are the constraint.
The constraints. Constraints are physiological quantities the simulator can compute. The first is joint excursion beyond the range of motion, which is also MyoChallenge's pain proxy. The second is muscle force above a set fraction of maximum isometric force while the muscle is lengthening, a rough proxy for strain injury. The third is human-device interaction torque above a comfort limit. For locomotion, a fall is added. Each has a per-session budget, and the primary safety metric is the total number of violations accumulated during adaptation, not violations at convergence.
The mismatch-gated shield. The method is trained in two phases. On the proxy population in simulation, we train three things: a task policy, a backup policy (maximal passive compliance with zero assistive torque, which is certified safe on the proxies), and a safety critic using Sim-to-Lab-to-Real's safety Bellman equation. We also train an ensemble of dynamics models of the human-plus-device system on proxy transitions.
On a new individual, the controller runs under the shield for a fixed interaction budget. After each step it compares the observed next state with the ensemble's prediction. A state-action pair is flagged as belonging to the estimated mismatch region $\hat B$ when the prediction error exceeds the ensemble's own spread by a threshold. This is the continuous stand-in for the statistical test in Provably Safe Sim-to-Real. Flagged data is used to fit a residual correction $r_\phi$, giving the hybrid model
$$\hat f(s,a) = f_{\text{sim}}(s,a) + \mathbb{1}[(s,a)\in \hat B]\, r_\phi(s,a),$$
where $f_{\text{sim}}$ is the proxy simulator's transition. The safety critic is then refit by rolling out in $\hat f$, and the shield's switching margin is raised only in states near $\hat B$. Everywhere else the proxy-trained critic is trusted and the task policy acts freely. The hypothesis that makes this worth building is that patient mismatch is localized. Spasticity matters only above a stretch-velocity threshold, and weakness matters only near peak loads. If so, cost should scale with the size of $\hat B$ rather than with the whole state space, which is the continuous analogue of the $|B|$ result.
Baselines
The naive reference points are two. The first deploys the proxy-trained policy frozen, with no adaptation. The second fine-tunes on each patient with unconstrained Soft Actor-Critic plus a reward penalty, which is how AR3n and A Human-Sensitive Controller handle safety. Together they show what current practice would do if it were used on patients.
The constrained-RL baselines are PPO-Lagrangian and its PID-damped variant from Safety Gymnasium's SafePO library, and the backward value function method of Constrained MDPs via Backward Value Functions, which keeps the cost budget satisfied throughout training rather than only at the end. They represent the view that a principled constrained objective is enough without a model of mismatch. For reference, Safety Gymnasium reports that PPO-Lag cut violations by 98% relative to PPO at a 45% reward cost on velocity-constrained tasks. The backward value function paper reports no scalar reward figures, and the file has no reported result on Lacuna for any of these methods on MyoSuite.
The fixed-shield baseline is the Sim-to-Lab-to-Real safety critic and backup policy trained on proxies and never updated. It reported a certified safety bound rising from 0.304 to 0.708 and 86.7% safety on real hardware, in a navigation domain. It isolates the core question of whether a shield learned on proxies carries over unchanged to patients.
The strongest competitor is domain randomization over the pathology parameters. The shield and policy are trained on a population that samples sarcopenia, fatigue and reflex severity from a prior, and the margin is tuned so that the worst sampled patient stays within budget. This baseline is the real threat because it uses the same pathology knowledge that our cohort encodes.
Two ablations isolate the key ingredient. The first keeps adaptation but replaces mismatch gating with a uniform margin increase of equal average size. The second keeps gating but drops the residual refit of the safety critic.
Every method gets the same simulated training budget, the same hyperparameter search budget tuned on proxies only, and the same per-patient interaction budget. That budget is set to the equivalent of a 10-minute session, matching the fine-tuning window in A Human-Sensitive Controller. Evaluation uses held-out patients whose parameters are drawn outside the training range as well as inside it. Beating the strongest baseline means matching randomized-worst-case violations within statistical error while achieving significantly lower tracking error and less over-assistance on mild and moderate patients, or equivalently a strictly better violations-versus-task Pareto front across severity levels.
Why not just…?
Why not just randomize pathology parameters in training and ship one robust shield? This is the most serious alternative, and it may succeed on violations. The expected weakness is conservatism. A shield tuned for the worst plausible spastic patient intervenes constantly on a mildly affected one, which is the "safe but useless" outcome and, for assist-as-needed control, the over-assistance that AR3n argues blocks motor learning. It also fails silently when a patient falls outside the prior. The severity-stratified comparison and the out-of-prior held-out patients will settle it. If randomization matches us on both violations and task quality, we will report that and not claim the method.
Why not just clamp torque and joint angle with a classical controller? Hard clamps prevent some violations, but Natural Multicontact Walking found that non-physiological model-based gaits raised user muscle effort, and Virtual Impedance Adaptation found fixed amplification control resistive across walking speeds. More to the point, a clamp tuned on proxies does not account for a reflex that produces its own torque, since the danger comes from the human's muscles and not the device. We include a clamped PD controller as an extra reference to test this directly.
Why not just penalize violations in the reward? Penalty tuning either permits violations or freezes the policy (Safe RL via Constrained MDPs). In the tabular sim-to-real setting, unconstrained transfer violated safety in 68.5% of learning iterations versus 0% for certified exploration (Provably Safe Sim-to-Real Transfer). The penalized Soft Actor-Critic baseline measures whether this carries over to musculoskeletal models.
Why not just collect data on real patients? Eventually that is necessary, and nothing here substitutes for it. But a simulated audit tells a clinical study which safety claims from proxy experiments are trustworthy before any patient is exposed to them. Patient sessions are short, and MyoSuite reports over 70 million samples for hard tasks, so learning from scratch on a person is not possible.
Proposed Experiments
The first decisive check is the audit. For every baseline, compute the proxy optimism gap: violations per session on patients minus violations on proxies, under the same method and budget, stratified by severity and by reflex model. The claim requires this gap to be large for methods that look safe on proxies, and requires the patient violations to fall inside the pathology-affected states. If the gap is negligible under both reflex models, the proxy trap does not matter for safe exploration in this regime, and that is itself a publishable negative result.
The second check is the head-to-head against domain randomization and the two ablations, using the Pareto criterion above. The uniform-margin ablation is the key comparison. If it performs as well as gating, localization is not what helps, and the method claim is withdrawn.
The third check tests the scaling hypothesis. Vary the number of pathological muscles and the reflex threshold, measure the fraction of visited state-action pairs flagged in $\hat B$, and test whether per-patient violations and the interactions needed to reach target tracking error grow with that fraction rather than staying constant. Finally, the whole pipeline is repeated on the myoOSL locomotion cohort with falls as the constraint, to check that the conclusions are not specific to the arm.
References
- Natural Multicontact Walking for Robotic Assistive Devices via Musculoskeletal Models and Hybrid Zero Dynamics
- AR3n: A Reinforcement Learning-Based Assist-as-Needed Controller for Robotic Rehabilitation
- A Human-Sensitive Controller: Adapting to Human Ergonomics and Physical Constraints via Reinforcement Learning
- Provably Safe Sim-to-Real Transfer
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- MyoSuite - A contact-rich simulation suite for musculoskeletal motor control
- MyoChallenge 2024: A New Benchmark for Physiological Dexterity and Agility in Bionic Humans
- Constrained Markov Decision Processes via Backward Value Functions
- Virtual Impedance Adaptation of Lower-Limb Exoskeleton for Human Performance Augmentation Based on Deep Reinforcement Learning
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning