Failure-Frontier Curricula for Recovering Synchronized Actuators

Machines that move one load with several motors, such as gantry cranes, shaftless printing presses, cable-laying trains and two-handed robots, stay safe by keeping their motors in step. Classical controllers do this well when nothing unusual happens. When one motor stalls, slips or suddenly lags, the only fallback is usually an emergency stop. Reinforcement learning (RL, learning a controller by trial and error against a reward) could in principle learn to recover instead. In practice it runs into two problems. Faults are rare within normal operation, so almost none of the experience an agent collects is about recovering. And severe faults are unrecoverable at first, so the agent gets no useful learning signal from them.

This note proposes to train a small learned correction on top of a classical synchronizing controller, using only fault episodes. Each training episode is reset to a snapshot taken just before a synthetic fault is injected. The faults are chosen by a curriculum that stays at the failure frontier: the fault severities and timings where the current controller recovers about half the time. The frontier starts where the classical controller alone begins to fail and moves outward as the learned correction improves. The claim is that concentrating experience on recovery, at the right difficulty, reduces the simulation frames needed to learn robust recovery by a large factor compared with injecting faults at random. We also claim this helps more than applying known curriculum methods to the fault parameters, and that it does not degrade normal synchronized tracking.

Core Claim and What Is New

Three ingredients already exist separately in the file. We combine them, and the combination is what we test.

The first ingredient is a classical base controller that handles normal synchronization. Cross-coupling control feeds the difference between two motors' tracking errors, $\varepsilon_{ij}=e_i-e_j$, back into each motor's loop, so that motors slow down or speed up for a lagging peer (Multiple Axes Synchronizing. System under Adaptive Feedforward Control; Research on Coordinated Control of Multi-PMSM for Shaftless Overprinting System). These controllers are precise under nominal conditions but have no response to hard stalls beyond stopping. We keep such a controller as the base and never replace it.

The second ingredient is the idea of using the base controller's own failures to decide where learning is needed. Hybrid Reinforcement Learning and Its Application to Biped Robot Control first observed where a linear controller failed and then recruited RL only in those regions. This cut the trials needed for stable walking from roughly 800 to 50. That paper motivates our approach. It used a fixed, one-shot map of failure regions on a three-link planar robot with lookup-table learning. We turn the failure boundary into a moving curriculum over a parameterized fault space for many coupled actuators.

The third ingredient is training on the repair rather than on the whole trajectory. AgentRefine synthesizes trajectories in which a language agent makes an error and then corrects it, and computes the loss only on the correction. Reflect-RL similarly adds perturbed "negative" moves with recovery explanations. Both work in text environments and rely on language-model reflection, which takes seconds per step. That latency rules them out for motors on millisecond control cycles. We borrow the principle of spending the learning signal on the recovery window. We drop the language reflection entirely: the learned correction is a small network that runs at the control rate.

The closest curriculum method is Self-Paced Deep Reinforcement Learning (SPDL). SPDL moves a Gaussian distribution over task parameters from easy tasks toward a fixed target distribution, trading expected value against divergence from that target. SPDL has two limitations here. It works on external task parameters in single-robot settings. And its unimodal Gaussian fits poorly with fault spaces that mix discrete choices (which motor, which fault type) with continuous ones (magnitude, onset phase). Our curriculum has no target distribution to approach. Its reference point is the base controller's failure boundary, and it starts every episode at the fault rather than at the start of the task. We treat SPDL as a baseline, not as a component.

Research Program

We define a fault as $\phi=(k, \text{type}, m, t_0)$. Here $k$ is the affected actuator, the type is stall, torque loss, slip or added lag, $m$ is the magnitude, and $t_0$ is the onset time within the nominal motion. Each motor's command is the cross-coupling controller's output plus a bounded residual from a shared learned policy. The policy sees local tracking errors, neighbours' synchronization errors and recent torques.

An episode is successful if all synchronization errors return below a tolerance within a time budget. The episode fails if any error exceeds a damage limit first. This limit stands in for a torn web, a tilted load or a dropped object.

We keep a running estimate $\hat s(\phi)$ of the current combined controller's success rate. The estimate is a small classifier fitted to recent outcomes, which lets it generalize across the mixed discrete and continuous fault space. The curriculum samples faults with probability

$$p(\phi)\propto \exp\!\left(-\frac{(\hat s(\phi)-\tfrac12)^2}{\sigma^2}\right),$$

where $\sigma$ sets the width of the difficulty band. We use this form because faults the controller always handles or always fails produce near-zero advantage signal in policy-gradient updates. The band concentrates samples where outcomes vary.

At initialization the residual is zero, so $\hat s$ is the classical controller's success rate. The first frontier is therefore exactly the fault region where cross-coupling control starts to break. This is the curriculum form of the hybrid-RL paper's two-step idea.

Each episode begins from a stored simulator snapshot a short window before $t_0$. The policy-gradient loss is computed only on steps after onset. A small penalty on the residual's size during the pre-fault window keeps nominal behaviour close to the classical controller. To guard against forgetting, a fixed fraction of the batch replays faults from earlier frontiers. This addresses the stage-to-stage forgetting that the file lists as a known curriculum limitation.

Training uses on-policy multi-agent PPO with one agent per actuator group, run in massively parallel GPU simulation. We choose on-policy training because MTBench found that under high simulation throughput, multi-task PPO reached good performance in 22 minutes on Meta-World MT10, versus 12 hours for multi-task SAC. The same benchmark reports that sparse-reward multi-task RL fails without a curriculum. Recovery tasks have exactly that sparse structure.

Testbeds

We use two testbeds. The first is a family of shared-load environments we build in Isaac Gym. Two to eight simulated motors drive a common load: a dual-drive gantry beam, a three-roll web-tension line modelled on the shaftless overprinting setup, and a chain of traction units modelled on Design of Intelligent Distributed Cable Laying System Based on Multi-machine Cooperative Control Strategy. Friction, backlash and a flexible link are randomized. Cross-coupling control is a natural, well-tuned baseline here, and actuator count can be scaled.

The second testbed is Bi-DexHands, two 24-degree-of-freedom hands simulated at over 30,000 frames per second across 2,048 parallel environments. We use its tasks in which both hands jointly hold or move one object. Faults are injected into individual finger or wrist actuators, and the base controller is the benchmark's trained multi-agent PPO policy rather than a classical controller. This second testbed checks whether "frontier relative to an existing controller" transfers when the base is itself learned.

Baselines

The naive reference point is the base controller alone with an emergency stop at the damage limit. This is today's industrial behaviour. The comparison measures how much recovery any learning adds at zero training cost.

The second baseline is multi-agent PPO trained end-to-end on normal episodes, with synchronization error added to the reward and no fault injection. This is the Bi-DexHands-style recipe, in which coordination emerges from reward alone. It tests the claim that coordination learned this way is fragile.

The strongest competitor is the same residual architecture trained with faults drawn uniformly from the full fault space, using identical reset-to-fault snapshots and the same post-onset loss. This is domain randomization with our data-concentration trick included. Every element of our method is present except the frontier curriculum. On the high-throughput simulators in the file, uniform randomization plausibly works well.

The fourth baseline is SPDL applied to the continuous fault parameters, with one SPDL distribution per discrete fault type and a target distribution uniform over the full space. This tests whether an existing self-paced method already captures the gain.

All learned methods get the same network, the same number of environment frames, the same parallel-environment count and the same hyperparameter search budget. That budget is twenty configurations per method, selected on a validation fault set. We run five seeds.

Beating the uniform-randomization baseline has a concrete meaning: reaching its final recovery rate on a held-out fault set in at most half the frames, or exceeding its recovery rate at equal frames by a margin outside seed variance. Nominal tracking error must not worsen significantly.

Two ablations isolate the key ingredients. The first removes reset-to-fault, so faults occur at random times within full episodes. This separates the effect of data concentration from the effect of the curriculum. The second defines the initial frontier by a from-scratch policy rather than by the classical controller. This tests whether anchoring on the base controller's failure boundary matters.

On reported numbers: the file has no reported recovery rate or synchronization-error result for any method on fault-injected multi-actuator tasks. No reported result was found on Lacuna for fault recovery on Bi-DexHands, the shaftless overprinting testbed or the cable-laying platform. The nearest reported figures come from settings that are not comparable. SPDL reaches an asymptotic reward of 9.35 ± 0.1 on 3D Point Mass, against 2.34 ± 0.2 for ALP-GMM (Self-Paced Deep Reinforcement Learning). Two-step hybrid RL needs 50 trials versus about 800 for standard RL on the stilt biped (Hybrid Reinforcement Learning and Its Application to Biped Robot Control). We will therefore establish all baseline numbers ourselves under the shared protocol.

Why not just…?

Why not just tune the classical cross-coupling controller better, or adapt its gains online? Gain adaptation handles slow parameter drift. It cannot produce qualitatively new behaviour, such as deliberately stalling the healthy motors to match a jammed one or redistributing load around a slipping unit. The hybrid-RL paper shows the same pattern: a linear controller that works on flat ground failed completely on a 10 cm slope. Our "base controller alone" and "adaptive cross-coupling" rows measure exactly how large the unrecoverable region is. If that region turns out to be small in our testbeds, the proposal loses its motivation, and we will report that.

Why not just randomize faults uniformly and rely on massive parallelism? This is the real threat. At 30,000 frames per second, throughput can compensate for poor sampling. The uniform-with-reset baseline gets every advantage except the curriculum. We expect it to lose most clearly as the fault space grows, meaning more actuators and more fault types, because the share of samples near the frontier shrinks. We will therefore plot the gap against actuator count from two to eight. If the gap does not grow with scale, we conclude the curriculum is unnecessary and that reset-to-fault concentration alone carries the gain.

Why not just run SPDL with fault parameters as its task context? SPDL is the principled version of "easy to hard" and is run as a full baseline. Our reasons for expecting it to underperform are specific. Its Gaussian distribution cannot represent a frontier that is scattered across actuators and fault types. And its pull toward a target distribution keeps spending samples on faults that remain unrecoverable. If SPDL matches our method, the contribution reduces to reset-to-fault training plus a residual on a classical controller, and we would say so.

Why not just add language-model reflection, as in Reflect-RL? Diagnosis by reflection takes seconds. A stalled motor on a coupled load does damage within milliseconds, so reflection cannot sit inside the recovery loop. At most it could help design fault taxonomies offline. We leave that out of scope.

Decisive Checks

The central result is a curve of held-out recovery rate against environment frames for all methods, on both testbeds. The main contrast is our method versus uniform randomization with reset. The same comparison is repeated across actuator counts to test whether the advantage grows with the size of the fault space.

Three secondary checks carry the other claims. The first is nominal tracking and synchronization error before and after training, which tests that the residual stays out of the way during normal operation. The second is recovery on fault types withheld from training, for example lag faults held out when training on stalls and slips. The third is the forgetting curve on early-frontier faults as the frontier advances.

A final measurement tracks how the learned frontier moves relative to the classical controller's initial failure boundary. It shows where learning actually extends recovery, which a practitioner would need to decide where an emergency stop is still required.