A redundant robot, such as a seven-joint arm or a humanoid, has more joints than its main job needs. Classical controllers use the spare joints for lower-priority goals by projecting those goals into the motions that leave the main task untouched. Reinforcement learning on these robots rarely uses this structure inside the gradient estimate. The exploration noise a policy injects into the spare joints barely affects the main task, yet the standard policy-gradient estimator still credits that noise against the main task's reward. The noise adds variance and carries almost no signal. We propose to split the policy's randomness along the robot's known kinematic geometry. The main-task gradient is then computed only from the noise that can move the main task, and the secondary-task gradient only from the noise in the spare joints. The aim is an estimator with lower variance and therefore better sample efficiency, together with strict priority enforced at the action level rather than by projecting noisy parameter gradients.
In two sentences: a policy for a redundant robot should explore separately in "task directions" and "spare directions," which the robot's Jacobian (the matrix mapping joint velocities to task-space velocities) defines exactly. Each objective's learning signal should then be paired only with the exploration that can actually affect it. This removes a known source of variance without having to estimate any extra structure from data.
Core Claim
Existing prioritized policy-gradient methods build the priority geometry from estimated gradients. LPPG-RL projects each lower-priority sample gradient onto the half-spaces defined by higher-priority sample gradients using Dykstra's algorithm. NSPO projects safety gradients into the null space of a capability matrix computed from calibration activations. In both cases the projector is a random quantity, so the noise in the primary estimate distorts the subspace into which the secondary update is pushed. The research file notes that no variance analysis of this compounding exists.
Our claim is that in robotics the relevant null space is not random. It is $P_1 = I - J_1^\dagger J_1$, computed analytically from the primary task's Jacobian $J_1(q)$, where $J_1^\dagger$ is its pseudoinverse. This is the same object used in operational-space control (Peters et al.). Concretely, the policy outputs a primary task-space command $u_1 \in \mathbb{R}^m$ and a null-space command $z_2 \in \mathbb{R}^{n-m}$ expressed in an orthonormal basis $N_1(q)$ of the null space. The executed joint command is
$$a = J_1^\dagger(q)\, u_1 + N_1(q)\, z_2, \qquad u_1 \sim \pi^1_\theta(\cdot \mid s), \quad z_2 \sim \pi^2_\theta(\cdot \mid s),$$
with the two Gaussians sampled independently. Because they are independent, the log-likelihood of an action splits into a primary part and a secondary part. We then use the estimators
$$\hat g_1 = \nabla_\theta \log \pi^1_\theta(u_1 \mid s)\, \hat A_1 + \beta\, \nabla_\theta \log \pi^2_\theta(z_2 \mid s)\, \hat A_1, \qquad \hat g_2 = \nabla_\theta \log \pi^2_\theta(z_2 \mid s)\, \hat A_2.$$
Here $\hat A_k$ is a per-objective advantage estimate (how much better than average the action was for objective $k$), and $\beta \in [0,1]$ controls how much null-space exploration is credited to the primary task.
When $\beta = 1$, the primary estimator is unbiased. When $\beta = 0$, it drops a term whose expectation is nonzero only because null-space motion changes the configuration $q$, and through it future primary performance. For locally linear kinematics that effect is second order. Dropping the term removes $n-m$ dimensions of exploration noise that would otherwise multiply $\hat A_1$, so for roughly isotropic exploration the primary score's variance should shrink by about the fraction $(n-m)/n$ of action dimensions. The secondary estimator never touches the primary head, so strict priority holds by construction at the action level.
Parameter-space projection, as in LPPG-RL, remains necessary only on any shared network trunk. That is a much smaller and better-conditioned problem.
The novelty is a specific estimator. It is a variance-reduction rule derived from known kinematic redundancy, linking the action-space null spaces of classical control to the parameter-space gradient variance of reinforcement learning. The research file identifies this link as unformulated (its Gap 2).
Relation to Prior Work
The priority idea comes from control. Peters et al. unify operational-space control laws under Gauss' principle and show that different weighting metrics give the same task accuracy with different joint motions. Task Space Inverse Dynamics makes such cascades fast and precise on a 23-joint humanoid. Both assume known models and do no learning. We reuse their projectors as fixed, exact parts of the policy's output layer.
Two learning papers motivate the project. Sharma et al. let a reinforcement-learning policy pick and order object-centric controllers composed through analytical null-space projection. They report over 98% success on Hex-Screw and Door-Opening, while end-effector reinforcement learning gets 0% under the same budget. This is strong evidence that kinematic structure buys sample efficiency, but the paper never analyses the gradient estimator. Composable Energy Policies uses a product of experts as a safety prior under PPO and also leaves estimator variance unstudied. Our question is whether part of these gains can come from the estimator rather than the action space alone, and whether the estimator adds gains on top of an already structured action space.
Three papers constrain the design. LPPG-RL reports "gradient vanishing" of lower-priority objectives when higher-priority constraints are active and patches it with extra on-policy rollouts (Subproblem Exploration). Our secondary head has its own exploration noise and its own gradient, so it cannot be starved by the primary head. Whether it is starved through the shared trunk is something we will measure. Skalse et al. prove convergence of Lagrangian lexicographic actor-critic methods. Safety-Gymnasium shows that Lagrange multipliers oscillate unless damped and that projection-based methods have lower variance. These results argue against building the method on dual variables.
HVD splits a whole-body critic additively by kinematic subtree and improves Wipe Board success from 18% to 32% offline. That is a value-side decomposition without priority. Ours is on the policy side, with strict priority and online learning.
Research Program
The work has three parts.
The first is analysis. For a linearized system with Gaussian exploration, we will derive the exact variance of the full and split primary estimators and the bias of $\beta = 0$ as a function of how quickly $J_1$ changes along null-space motion. The output is a computable quantity that predicts when splitting helps. Near kinematic singularities, $J_1^\dagger$ is replaced by a damped pseudoinverse and the null-space basis by a thresholded SVD. The analysis will track how the damping leaks primary-task effect into the "null" directions, and therefore how bias grows near singular poses.
The second is the algorithm. It is PPO with per-objective critics, the factored action head above, the split estimators, and an online choice of $\beta$. That choice compares the empirical variance reduction against a bias proxy obtained from occasional $\beta = 1$ batches. For more than two priority levels, the construction recurses using stacked Jacobians.
The third handles priorities that are not kinematic, such as "don't fall," which has no Jacobian. We use a kinematic surrogate, the centre-of-mass and support Jacobian, and test honestly whether the surrogate's null space is close enough to the true "irrelevant to survival" directions to keep the bias small.
Baselines
The naive reference point is PPO on a linearly scalarized reward with a plain joint-space Gaussian policy. Its weights are grid-searched within the same tuning budget as every other method. Scalarization is what practitioners actually do, and it sets the floor.
The strongest existing method for strict priorities in continuous control is LPPG-RL. It is the only method in the file that does lexicographic policy gradients with projection on MuJoCo Humanoid. We will run it with its Subproblem Exploration schedule, the same network sizes, the same number of environment steps and the same hyperparameter-search budget. It gets per-objective critics exactly as we do. The paper reports only qualitatively that it beats PPO on Humanoid; no reported result found on Lacuna for LPPG-RL Humanoid return. We will therefore reproduce it and treat our reproduction as the reference. Beating it means reaching its final primary and secondary returns in fewer environment steps, or attaining a higher secondary return at equal primary return, across at least five seeds with non-overlapping confidence intervals.
The strongest structured-action competitor is the Sharma et al. composition scheme. On Hex-Screw and Door-Opening it reports over 98% success against 0% for end-effector reinforcement learning. We will rerun it on those tasks with matched samples. Because it is already near ceiling on success, our comparison there is on environment steps needed to reach 95% success.
We will also run PPO-Lagrangian from the SafePO library. Safety-Gymnasium reports that PPO-Lag reduces violations by 98% relative to PPO in velocity tasks at the cost of a 45% drop in total reward. Matching its violation rate with a smaller reward drop counts as beating it.
The key ablation isolates the estimator. It uses the identical factored null-space action head and identical per-objective critics but the full score function, which corresponds to $\beta = 1$ with the secondary objective also crediting the primary head and projected LPPG-style in parameter space. Any gap between this ablation and our method is attributable to score splitting alone, not to the structured action space.
Why not just…?
Why not just put the null-space projection in the action layer and run ordinary PPO? This is the most serious threat. Sharma et al. suggest that most of the sample-efficiency gain may come from the action parametrization itself. Our key ablation is exactly this alternative. If score splitting adds nothing over the structured head with full scores, measured both as gradient variance per batch and as steps to threshold, the estimator contribution is null and we will report that. We expect a gap in proportion to the null-space dimension $n-m$, so it should be small on a 7-joint arm reaching a 6-D pose and large on a humanoid tracking a few task coordinates. The experiments vary this ratio deliberately.
Why not just use bigger batches or per-objective critics with generalized advantage estimation? Bigger batches cut variance at a linear cost in samples, which is the cost we are trying to avoid. Our metric is environment steps, so a batch-size sweep of the baselines is included and plotted on the same axis. Per-objective critics reduce the variance of $\hat A_1$ but cannot remove the factor of irrelevant null-space noise in the score term that multiplies it. Both baselines and our method get identical critics, so any remaining gap reflects that factor.
Why not just smooth the LPPG-RL projector, for example with a moving average of the primary gradient? A smoothed projector reduces noise in the subspace estimate but adds lag in a non-stationary problem, and it still learns from data a geometry that kinematics gives exactly. We include an exponential-moving-average variant of LPPG-RL as an extra baseline. This alternative competes only where no Jacobian exists, which is why the survival-surrogate study matters.
Why not just use Lagrangian constraints? Safety-Gymnasium documents multiplier oscillation and a 45% reward drop for PPO-Lag in velocity tasks. Skalse et al. note that threshold choice is brittle. We still run PPO-Lag with PID damping as a fair comparison rather than dismissing it.
Proposed Experiments
The first decisive check is a variance measurement before any learning curve. At frozen policy checkpoints we will estimate the per-coordinate variance of $\hat g_1$ and $\hat g_2$ from many independent batches under the full and split estimators. We will compare them to the analytical prediction and report bias as the cosine and norm gap to a high-sample $\beta = 1$ reference gradient. If measured variance reduction does not track $(n-m)/n$, or if bias near singular poses dominates, the core claim fails at its root.
Learning experiments then run on the KUKA-LWR reactive benchmarks from Composable Energy Policies, with the primary objective being collision-free end-effector progress and the secondary objective null-space posture or clearance. The reference result there is CEP's 46% success in Cage II against 4% for additive potential fields. They also run on the Franka Hex-Screw and Door-Opening tasks of Sharma et al., and on MuJoCo Humanoid with LPPG-RL's ordering of survival, then speed, then energy, using the centre-of-mass surrogate Jacobian for the top level. Metrics are environment steps to a fixed primary-return or success threshold, final secondary return at matched primary performance, primary-constraint violation rate during training, and seed-to-seed spread of learning curves.
The cleanest test of the mechanism is a controlled sweep of redundancy on a planar arm. It holds the task fixed and varies the joint count from 3 to 10. Our account predicts that the split estimator's advantage over the key ablation grows with $n - m$ and vanishes when $n = m$. A flat curve would falsify the account even if the method wins elsewhere for other reasons.
References
- LPPG-RL: Lexicographically Projected Policy Gradient Reinforcement Learning with Subproblem Exploration
- Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization
- A unifying framework for robot control with redundant DOFs
- Prioritized motion-force control of constrained fully-actuated robots: "Task Space Inverse Dynamics"
- Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation
- Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning
- Lexicographic Multi-Objective Reinforcement Learning
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Hierarchical Value-Decomposed Offline Reinforcement Learning for Whole-Body Control