A robot arm commanded in task space (where the hand should go) must convert that command into joint motions by inverting the Jacobian, the matrix that maps joint speeds to hand speeds. Near a kinematic singularity, such as a fully stretched arm, this inversion becomes ill-conditioned. A one-millimetre error in where the robot thinks its target or its own joints are can then demand a large, fast joint reconfiguration. Simulation-trained policies are usually developed with perfect state and then hardened with domain randomization, meaning random perturbation of physics and sensor parameters during training. These policies have no direct reason to avoid such configurations. The critic that scores them during training, the network that estimates future return, sees clean state, and the resulting failures appear only on hardware as velocity faults, torque spikes, and jerky tracking.
This note proposes to measure that hazard directly in simulation and constrain it during training. At every simulated step the low-level controller is run twice, once on the noisy observation the robot would actually receive and once on the true state. The size of the difference between the two joint commands becomes the cost in a constrained reinforcement learning problem. Because the cost depends jointly on how close the arm is to a singularity and on how noisy the sensors currently are, the policy can learn a singularity margin that grows with noise and shrinks to nothing when sensing is clean. This avoids the "safe but useless" over-conservatism of a fixed manipulability penalty.
Core Claim
The quantity that matters at deployment is not distance to the singular set but sensor-induced command error. That error is the product of how badly conditioned the controller's inverse is at the current pose and how large the perception error is. In simulation this product can be computed exactly, as a privileged counterfactual. Let $u(\cdot)$ be the low-level task-to-joint controller, for example a damped least-squares inverse. Let $s_t$ be the true state and $\hat s_t$ the noisy observation drawn from the randomized sensor model. The proposed per-step cost is
$$c_t = \max\big(0,\ \|u(\hat s_t) - u(s_t)\|_W - \tau\big),$$
where $W$ weights joints by their velocity or torque limits and $\tau$ is a tolerated deviation. To first order, $\|u(\hat s)-u(s)\| \approx \|J_\lambda^{+}(q)\|\,\|\delta x\|$, where $J_\lambda^{+}$ is the damped inverse, $\lambda$ the damping, and $\delta x$ the perception error. This makes explicit that the cost couples conditioning and noise. With damping, the gain $\sigma/(\sigma^2+\lambda^2)$ along each singular direction is capped at $1/(2\lambda)$. That cap is why damping alone forces a trade between noise amplification (small $\lambda$) and tracking error (large $\lambda$). A policy that keeps margin only when noise is high lets the controller run with small damping the rest of the time.
The training objective is a constrained Markov decision process: maximize task return subject to an expected discounted budget on $c_t$. It is solved with an off-the-shelf first-order constrained optimizer. The critic for the cost receives the sampled noise parameters as privileged input. Without that input, the critic cannot tell why the same pose is cheap in one episode and expensive in another.
Relation to Prior Work
Several works motivate the gap. Decoupling Skill Learning from Robotic Control trains a floating-gripper policy and lets a whole-body quadratic-program controller handle joint limits and singularities on a 13-DOF mobile manipulator, but it assumes noise-free perception and leaves sim-to-real as future work. Our cost is designed for exactly the noisy-input regime that work sets aside. Asymmetric Actor Critic gives the critic privileged state while the actor sees noisy images, and transfers zero-shot to a 7-DOF Fetch arm. However, its privileged critic is precisely what hides the noise-induced cost, and we extend the privileged side to include the noise realization itself. Navigation and prediction studies such as Changan Chen et al., Haoran Song et al. and Kaleb Ben Naveed et al. document how planners trained on clean state degrade under realistic sensor noise. None of them ties the degradation to Jacobian conditioning.
Other work constrains the design. Singularity-Avoidance Control with Model Mismatch and Actuator Constraints builds a barrier-function safety filter with Gaussian-process mismatch bounds on a 2-DOF planar arm. It shows that unguided control drifts into the singular region under a hidden payload, which argues against relying on damping alone. That filter assumes deterministic state observations and a known model, so it is a natural runtime complement rather than a competitor for the learned policy. The reward-shaping failure modes described by Quanyi Li et al. rule out folding the cost into the reward with a hand-tuned weight. The tuning-heuristics study How to pick the domain randomization parameters warns that broad randomization makes policies sluggish. Our cost is meant to concentrate conservatism where conditioning and noise jointly demand it, instead of spreading it everywhere.
Two papers supply methods. FOCOPS provides the first-order constrained policy update, and PPO-Lagrangian from the SafePO suite in Safety Gymnasium provides a second solver. Provably Safe Sim-to-Real Transfer is the closest work joining safe exploration with sim-to-real. It works in tabular gridworlds and guarantees zero violations during real-world learning, whereas we target continuous high-DOF arms with zero real-world training and make no formal guarantee. The STEEL paper's "singularities" are gaps in statistical support, not Jacobian rank loss, and are unrelated.
Research Program
The work has three parts. The first is a simulation environment in ManiSkill/SAPIEN (ManiSkill) with a 7-DOF arm and a damped least-squares Cartesian controller. Its randomized sensor model covers joint-encoder noise and bias, camera extrinsic offsets, depth noise on the target estimate, and latency. The magnitude of each is sampled per episode, alongside standard dynamics randomization of friction, mass and gains as in Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. The counterfactual controller call adds one extra inverse per step, which is cheap.
The second part is the training recipe. The actor receives noisy observations. The reward critic receives true state. The cost critic receives true state plus the sampled noise parameters. Optimization uses FOCOPS and PPO-Lagrangian.
The third part is an analysis of what the policy learns. We will plot the distribution of minimum Jacobian singular value visited against the noise level sampled for the episode. The central hypothesis predicts a monotone relationship. A fixed manipulability constraint predicts a flat one.
Tasks are chosen so that near-singular poses are useful, not avoidable for free. They are reaching to targets at the workspace boundary, the OpenCabinetDrawer task (the drawer must be pulled toward full extension), and pick-and-place with targets near arm extension. Hardware evaluation is zero-shot on a 7-DOF arm with an RGB-D camera, in the style of the Fetch setup.
Baselines
The naive reference is PPO trained with domain randomization of the same sensors and dynamics, the same damped controller, and no constraint. It answers whether randomization alone teaches the policy to keep margin.
The strongest existing system on the manipulation question is the decoupled skill-plus-QP approach of Kai Lu et al.. It reports a 51% success rate on unseen test cabinets in OpenCabinetDrawer (74% on training cabinets), against 10–13% for end-to-end SAC and behavior cloning, all with clean state. We will rerun it on our randomized-noise version of the same 25 cabinets (15 train, 10 test), with its QP singularity terms intact and the same simulation-step budget. Beating it means matching its clean-state success within noise while showing higher success and fewer velocity or torque limit violations once sensor noise is injected. No reported result was found on Lacuna for this method under sensor noise, so that number is ours to produce.
For zero-shot hardware transfer, the reference is the Asymmetric Actor Critic setup, which reports 5/5 real-world successes on block tasks. We will run its asymmetric training with our randomization as a baseline. The safety solver itself is calibrated against the Safety Gymnasium report that PPO-Lag cut violations 98% relative to PPO at a 45% reward cost.
Three ablations isolate the key ingredient. The first constrains a fixed manipulability threshold, Yoshikawa's index $\sqrt{\det(JJ^\top)}$, which is noise-blind. The second constrains raw joint-velocity and torque-limit exceedances under noisy rollouts, which is noise-aware but not counterfactual. The third uses our cost but withholds noise parameters from the cost critic. All arms share the controller, randomization ranges, network sizes, simulation steps, and an equal hyperparameter search budget of the same number of seeds and configurations per method.
Why not just…?
Why not just use damped least squares and tune the damping? This is the most serious alternative. Damping caps the noise gain at $1/(2\lambda)$, but only by inducing tracking error, and tracking error matters when the gripper holds a rigid handle. Singularity-Avoidance Control also shows unfiltered control drifting into the singular region under mismatch. We settle the question empirically. We sweep $\lambda$ for the unconstrained policy, including an adaptive damping rule that scales with estimated noise, and trace the frontier of tracking error against noise-induced command error. The proposal must lie strictly below that frontier. If it does not, damping suffices and the idea is wrong.
Why not just put a manipulability constraint in the constrained problem? It does stop singular configurations. But it charges the same price whether sensors are clean or noisy, so it either over-restricts boundary reaching under clean sensing or under-protects under heavy noise. This is the first ablation. The decisive measurement is success on boundary targets at low noise together with spike rate at high noise.
Why not just constrain joint-velocity and torque-limit violations directly? This is a real threat, because it is simpler and also noise-aware. Our argument is that a violation signal is sparse and confounds fast, task-driven motion with noise-driven jitter. The counterfactual difference is dense, nonzero before any limit is hit, and attributes cost specifically to perception error. If the second ablation matches ours on hardware fault counts and success, we will report that the simpler constraint is enough.
Why not just filter the sensors? A Kalman-style filter shrinks $\delta x$ but leaves the conditioning term untouched, and it adds lag that randomization must then cover. We include filtering as a layer applied to all methods rather than as a substitute.
Proposed Experiments
In simulation we will report task success, peak and 99th-percentile joint velocity and torque, the count of limit trips, end-effector tracking error, and the noise-versus-minimum-singular-value curve. These are measured across a grid of injected noise levels that includes levels outside the training range, to test out-of-distribution behavior.
On hardware we will report zero-shot success on boundary reaching and drawer opening, along with controller fault events and peak joint speeds logged at the drive level. We will compare the proposal, the decoupled-QP baseline, the asymmetric actor-critic baseline, and the three ablations over a fixed number of trials per method and task, with targets and noise conditions randomized in a common order.
The decisive checks are three. First, the learned singularity margin should rise with noise level, which neither the manipulability ablation nor the naive policy should show. Second, the proposal should dominate the damping frontier rather than sit on it. Third, on hardware it should reduce limit trips relative to the velocity-limit ablation at equal success. Failing the second check falsifies the core claim. Failing only the third means the counterfactual cost is an elegant but unnecessary refinement.
References
- Decoupling Skill Learning from Robotic Control for Generalizable Object Manipulation
- Asymmetric Actor Critic for Image-Based Robot Learning
- Crowd-Robot Interaction: Crowd-Aware Robot Navigation With Attention-Based Deep Reinforcement Learning
- PiP: Planning-Informed Trajectory Prediction for Autonomous Driving
- Trajectory Planning for Autonomous Vehicles Using Hierarchical Reinforcement Learning
- Singularity-Avoidance Control of Robotic Systems with Model Mismatch and Actuator Constraints
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?
- First Order Constrained Optimization in Policy Space
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Provably Safe Sim-to-Real Transfer
- ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots