Many machines with several motors are controlled by first "untangling" them. A fixed matrix, or a coordinate change, converts commands so that each output responds to exactly one input. Once that holds, a single-input single-output (SISO) loop, such as a simple PID controller, can run each channel independently and very fast on a cheap microcontroller. The untangling matrix is computed from nominal physical parameters such as mass, inertia, or inductance. When those parameters drift, the channels re-tangle and the fast scalar loops start fighting each other. This is the sim-to-real problem in miniature. A real robot differs from its model in exactly the parameters the decoupler depends on.
We propose to keep the fast SISO loops and the analytical decoupler, and to learn only a small, slowly updated correction to the decoupling matrix. The correction is computed from a short window of recent inputs and outputs. It is trained in simulation under domain randomization, which means training across many randomly perturbed copies of the plant so that the real one looks like just another sample. The bet has three parts. Learning a correction to a structured matrix needs far fewer samples than learning a residual over all motor commands. The correction can run at a fraction of the control rate, so it adds nothing to the latency of the fast loop. And it targets the specific failure mode, re-coupling under parameter mismatch and saturation, that makes decoupled controllers brittle.
Core Claim
Decoupling transforms such as precompensator matrices, generalized-inverse force allocation, and d-q current projection all fail in the same way. If the true plant is $G = G_0 + \Delta G$ and the nominal decoupler is $A_0^{-1}$, the loops see $G A_0^{-1} = I + \Delta G\, A_0^{-1}$ instead of the identity. The off-diagonal part of $\Delta G\, A_0^{-1}$ is cross-talk that the SISO loops were never tuned to handle. We replace the fixed decoupler with
$$u_t = \big(A_0^{-1} + \Phi_\theta(h_t)\big)\, v_t ,$$
where $v_t$ is the vector of outputs of the independent SISO loops and $h_t$ is a window of past commands and measurements. $\Phi_\theta$ is a small network producing a low-rank matrix correction, and $u_t$ is the actuator command. $\Phi_\theta$ is evaluated at a slow rate, roughly 10–50× slower than the SISO loops. This is reasonable because parameter drift and contact-induced coupling change slowly compared with the inner control bandwidth.
The claim is that this parameterization is better than a generic additive residual on $u_t$ in three ways, and that each can be measured. It is more sample-efficient, because the output has rank-limited structure tied to the coupling. It is cheaper at run time, because nothing learned sits on the fast path. It degrades more gracefully under actuator saturation, because the correction can learn saturation-aware reallocation that the analytical inverse cannot express.
Where This Sits
The decoupling literature supplies the base mechanisms and states their weakness. Precompensators diagonalize a distillation column (Raisch) and split a disturbance-free-payload spacecraft into 12 PID loops (Jin et al.). Generalized inverses split maglev planar-motor winding currents into force and torque channels (Kou et al., Xing et al.). Geometric splitting runs two small MPCs, one for the horizontal and one for the vertical plane of an underwater vehicle, on embedded hardware (Yang et al.). All of these assume nominal parameters and are evaluated without parameter randomization. That untested assumption is exactly where our work starts.
The closest competing idea treats coupling as a per-channel disturbance and cancels it with an extended state observer, as in the maglev ruler controller (Levitating Control System of Maglev Ruler Based on Active Disturbance Rejection Controller). It is classical, sample-free, and a serious baseline. It differs from us in that each observer sees only its own channel, whereas $\Phi_\theta$ sees all channels jointly.
Residual learning supplies the training method and motivates the sample-efficiency argument. Residual Reinforcement Learning for Robot Control adds a TD3-trained correction to a hand controller. Residual Policy Learning adds residuals over MPC or scripted policies and shows that zero-initializing the output layer plus a critic burn-in protects the base controller. Hybrid Reinforcement Learning and Its Application to Biped Robot Control cut trials from 800 to 50 by restricting learning to where the linear controller fails. We reuse zero-initialization, so $\Phi_\theta = 0$ recovers the nominal decoupler exactly. All three of these works place the residual additively in the full, coupled action space. We place it multiplicatively inside the decoupler, so the learned object is a coupling estimate rather than a command.
The sim-to-real work supplies the evaluation regime and constrains the design. Sim-to-Real: Learning Agile Locomotion For Quadruped Robots shows that dynamics randomization plus explicit injection of the 15–19 ms onboard latency is needed for zero-shot transfer. It uses an additive policy over an open-loop gait in the coupled actuator space. RRLS reports that broad randomization buys average-case robustness at the cost of nominal performance. This is the robustness–performance dilemma, and it is a warning for us: if $\Phi_\theta$ is trained over too wide a range it may learn a conservative average correction. Feeding it the history $h_t$, so that it can identify the current plant, is our response, and we test whether that response is enough.
Research Program
The work has four parts.
Plant suite. We build three coupled plants in simulation, each with a published analytical decoupler.
- A maglev planar-motor model with generalized-inverse allocation, following Kou et al. and Xing et al.
- A six-degree-of-freedom underwater vehicle with the horizontal/vertical split of Yang et al.
- The Minitaur quadruped in PyBullet with Tan et al.'s randomization ranges and latency model. Here we treat each leg's two motors as a coupled 2×2 system and use a fixed leg-coordinate transform (swing and extension) as the nominal decoupler.
For each plant we define randomization over exactly the parameters the decoupler depends on, namely masses, inertias, coil constants, and hydrodynamic coefficients. We also randomize actuator limits, so that saturation episodes occur.
Correction training. $\Phi_\theta$ is trained in two stages. First, a supervised stage regresses the correction implied by the true randomized parameters, $A^{-1}(\xi) - A_0^{-1}$, from history. Simulation gives us privileged access to $\xi$, the true parameter vector. Second, an RL stage fine-tunes on closed-loop tracking cost, which lets the correction depart from the analytical inverse where that inverse is not optimal, chiefly under saturation.
Systematic sweeps. We sweep the rank of the correction, the update-rate ratio, and the history length.
Hardware deployment. The SISO loops run on the microcontroller and $\Phi_\theta$ runs on a companion processor. We measure end-to-end timing to confirm that the fast loop's period is unchanged.
Baselines
The naive reference is the nominal decoupler with tuned SISO loops and no learning. It is the floor every method must clear under randomized parameters. Its nominal-parameter performance is the ceiling we hope to approach on the real plant.
The strongest learning competitor is an additive residual policy in the coupled actuator space, trained with the same randomization and latency injection, following Tan et al. and Johannink et al. On Minitaur it is literally the Tan et al. setup. For that setup the file reports a real-world galloping speed of 1.18 m/s against 1.34 m/s in simulation (Sim-to-Real: Learning Agile Locomotion For Quadruped Robots). On the maglev and underwater-vehicle plants, no reported result was found on Lacuna for learned residuals under parameter randomization, so we establish that number ourselves.
Fairness is enforced in three ways:
- The residual baseline gets the same simulator, the same randomization distribution, and the same environment-step budget.
- It gets the same hyperparameter-search budget, a fixed number of configurations per method.
- It gets the same history input as our method.
Beating it means two things. First, reaching its final tracking error with at least 3× fewer samples. Second, matching or exceeding its real-hardware performance (speed and sim-to-real gap on Minitaur, tracking error on the others) while adding no learned computation to the fast loop.
The strongest classical competitor is per-channel active disturbance rejection with extended state observers, following the maglev ruler controller. It is tuned with the same number of trials we give to RL hyperparameters. Beating it means lower cross-channel interference under large mismatch and saturation, where single-channel observers cannot reallocate effort across actuators.
Two ablations isolate the key ingredient:
- Identical networks that output an additive command residual instead of a decoupler correction, at both fast and slow rates. This separates the effect of where the correction enters from the effect of how often it runs.
- Supervised-only versus supervised-plus-RL training, which tests whether closed-loop learning adds anything beyond parameter estimation.
Why not just…?
Why not just run residual RL in the full action space with domain randomization, as Tan et al. did? This is the main threat. It already transfers zero-shot, and Johannink et al. report 75% success where the base controller manages 10%. Our argument is that its output dimension equals the actuator count and it must run at the control rate, whereas $\Phi_\theta$ has low rank and runs slowly. That argument is only worth something if the sample-efficiency curves and the slow-rate ablation show it. If the additive residual matches us at equal samples and also tolerates slow updates, our contribution shrinks to an interpretability argument, and we will report it as such.
Why not just identify the parameters online and recompute the analytical inverse? This is what our supervised stage amounts to, so the comparison is built in. If supervised-only matches supervised-plus-RL everywhere, the honest conclusion is that a learned identifier is enough. Even then, the result would be useful, because it would show that randomized simulation training produces a working identifier for decoupled control. We expect the gap to appear in saturation episodes. There the analytical inverse no longer describes the best allocation, which is a known breakdown of decoupling under actuator limits.
Why not just treat coupling as a lumped disturbance per channel with an extended state observer? This approach needs no learning and works on maglev hardware. Its limit is structural: each channel can only push back on its own disturbance, and cannot shift effort to a less-loaded actuator. The interference and saturation tests below are designed to show whether that limit matters in practice.
Proposed Experiments
On each simulated plant we measure four quantities:
- Tracking error.
- An interference ratio: the response of each channel to sinusoidal excitation injected into the others, measured at the off-diagonal frequencies of interest.
- The sample count needed to reach a target error.
- The number of instabilities during episodes with randomized actuator saturation.
Each quantity is reported across 10 seeds, following the RRLS protocol. We evaluate on in-distribution randomized parameters and on a held-out range 1.5× wider, which probes the robustness–performance dilemma.
On Minitaur hardware we report forward speed, the sim-to-real speed gap against the 1.18/1.34 m/s reference, and fast-loop timing jitter with and without the learned component running.
Three checks decide the outcome.
- Sample efficiency. If decoupler correction does not beat additive residuals in sample efficiency at matched final performance, the core claim fails.
- Update rate. If slowing $\Phi_\theta$ to one-tenth the control rate costs more than a small fraction of tracking error, the latency argument fails.
- Value of RL. If supervised-only training matches RL fine-tuning under saturation, the reinforcement learning stage is unnecessary and the method should be presented as learned identification.
References
- Multivariable controller design for an industrial distillation column
- Disturbance Attenuation and Pointing Control System Design for an Improved Disturbance-Free Payload Spacecraft
- Improved ADRC for a Maglev Planar Motor with a Concentric Winding Structure
- Toward Reducing Undesired Rotation Torque in Maglev Permanent Magnet Synchronous Linear Motor
- Energy-optimal Three-dimensional Path-following Control of Autonomous Underwater Vehicles under Ocean Currents
- Levitating Control System of Maglev Ruler Based on Active Disturbance Rejection Controller
- Residual Reinforcement Learning for Robot Control
- Residual Policy Learning
- Hybrid Reinforcement Learning and Its Application to Biped Robot Control
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- Rrls: Robust reinforcement learning suite