Domain-Conditioned Shadow Prices for Multi-Constraint Sim-to-Real Transfer

Robots trained in simulation usually face several hard limits at once: do not collide, do not exceed joint torque, do not overheat the motors, do not drain the battery too fast. Domain randomization trains the policy across many randomly perturbed versions of the simulator's physics (masses, friction, damping) so that the real robot looks like just one more sample. Current constrained versions of this recipe ask that each limit hold on average across the randomized physics. That is the wrong target. The real robot is a single physics setting, and it needs every limit to hold there simultaneously. Different limits also tend to fail under different physics. Low friction makes the robot slip into obstacles. A heavy payload makes it saturate torque. A policy can satisfy every averaged constraint and still break some limit in a large part of the randomization range.

We propose to enforce each constraint separately in every randomized physics setting, and to do so cheaply. We attach to each constraint a shadow price, meaning a penalty weight that rises when that constraint is violated. Instead of using one price per constraint, the price is a small learned function of the simulator parameters. During training, the function learns where in physics space each constraint binds, and the policy is penalized more heavily there. Because the simulator parameters are visible during training but not at deployment, they enter only the price function and a privileged critic. The deployed policy sees only its own sensor history.

Core Claim

Write $\xi$ for a vector of simulator physics parameters drawn from the randomization box $\Xi$, $R(\pi,\xi)$ for the expected return of policy $\pi$ in physics $\xi$, and $C_i(\pi,\xi)$ for its expected cumulative cost on constraint $i\in\{1,\dots,m\}$ with budget $d_i$. The problem we want to solve is the per-domain program

$$\max_\pi \; \mathbb{E}_{\xi}\left[R(\pi,\xi)\right] \quad \text{s.t.} \quad C_i(\pi,\xi)\le d_i \;\; \text{for all } \xi\in\Xi,\; i=1,\dots,m,$$

whose Lagrangian uses one nonnegative multiplier function per constraint, $\lambda_i(\xi)$:

$$\mathcal{L}(\pi,\lambda)=\mathbb{E}_{\xi}\Big[R(\pi,\xi)-\sum_{i=1}^m \lambda_i(\xi)\big(C_i(\pi,\xi)-d_i\big)\Big].$$

Three standard recipes are special cases of this Lagrangian. Making each $\lambda_i$ a constant gives the usual averaged-constraint Lagrangian under domain randomization. Collapsing the $m$ costs into one gives the scalar formulations used by current zero-shot safety methods. Freezing the weights gives monolithic reward penalties.

Our claim is that letting $\lambda_i$ vary with $\xi$ is the missing ingredient whenever the binding constraint changes across the randomization box. We expect it to produce policies whose joint feasibility, meaning the fraction of physics settings in which all $m$ constraints hold at once, is substantially higher than that of averaged or scalarized methods at matched task reward. We also expect it to avoid the uniform over-conservatism that a single pessimism coefficient induces. The learned map $\xi\mapsto\lambda(\xi)$ is a useful diagnostic in its own right. Regions where some $\lambda_i$ saturates at its cap mark physics settings where the constraints are jointly infeasible. Those regions tell the practitioner how far the randomization box can be widened before it forces sluggish behavior. That trade-off between too wide (conservative) and too narrow (brittle) is currently set by hand.

Relation to Prior Work

The closest work is SPiDR. It trains with domain randomization and adds a penalty proportional to the disagreement of an ensemble of randomized dynamics models, $\tilde c=c+\lambda\upsilon(s,a)$, to a single safety cost. On real hardware it reports satisfying constraints in all tests, while plain randomization produced 26.1x more violations on a Unitree Go1 and 6.2x more on a RaceCar. SPiDR motivates this proposal: it shows that randomization without constraint handling is unsafe on hardware. It differs from our approach in three ways. It handles one scalar cost. Its pessimism is a single hand-tuned coefficient that the authors note can freeze the policy. Its penalty depends on predictive variance, not on which constraint actually binds in which physics. Our method keeps a separate price per constraint and lets each price depend on $\xi$, so pessimism is spent only where a particular constraint is at risk.

Two constrained-RL papers supply the base machinery and also constrain our design. Central Path Proximal Policy Optimization (C3PO) replaces oscillating Lagrange multipliers with an exact ReLU penalty that tracks the interior of the feasible set. It is evaluated only on nominal, single-cost Safety Gymnasium tasks with a budget of 25.0, so its behavior under randomized dynamics is untested. Safety Gymnasium provides the tasks and the SafePO implementations of PPO-Lagrangian and PID-controlled multipliers. It reports that PPO-Lag cut violations by 98% at a 45% reward cost in velocity tasks, which illustrates how blunt a single global multiplier can be. We reuse C3PO's penalty as an optional stabilizer for the per-domain update rather than inventing a new one.

The dual pacing result of Robust Budget Pacing with a Single Sample supplies the intuition that shadow prices can be tracked from very little data. Dual FTRL attains near-optimal $\tilde O(\sqrt T)$ regret with one sample per period, though only for a single budget with independent rounds. Our price network is updated from one rollout per sampled physics setting. Smoothness of the network across $\xi$ stands in for repeated samples. The combinatorial work of Chekuri et al. and Badanidiyuru et al. motivates keeping constraints separate rather than summing them. Their barrier and contention-resolution analyses show that collapsing coupled budgets into one scalar leads to premature exhaustion of individual resources. We borrow that lesson, not their algorithms.

On the transfer side, Asymmetric Actor Critic supplies the trick of giving the critic privileged simulator state while the deployed actor sees only sensors. It achieved 5/5 zero-shot real manipulation successes. Here the privileged input is $\xi$, fed to the per-constraint cost critics and to the price network. Sim-to-Real: Learning Agile Locomotion and RRLS establish that plain randomization is a strong recipe for return. RRLS found it beat minimax adversarial methods such as RARL on average-case return. That finding constrains our design: we keep physics sampling non-adversarial and put all the reweighting into the prices, avoiding the unstable moving training target that RRLS observed for adversaries. Provably Safe Sim-to-Real Transfer gives zero-violation guarantees, but only for tabular single-constraint problems with real-world data collection and a known safe policy. We target zero-shot continuous control, so we treat it as context, not as a baseline. Dag et al. show on a Franka arm that a monolithic penalized reward transferred poorly (30% success, 70% crashes) compared to decoupled controllers. That is further evidence against scalarization.

Research Program

The method is a PPO-based learner with four components.

The actor conditions on a short history of observations and actions, never on $\xi$, so that it can implicitly infer the physics at deployment.

There are $m$ cost critics, plus one reward critic, all receiving the state and $\xi$.

A small price network $g_\phi(\xi)\to\mathbb{R}^m_{\ge 0}$ is trained by projected gradient ascent on the per-domain violation estimate $\hat C_i(\pi,\xi)-d_i$. That estimate comes from the episode rolled out in $\xi$. The network uses weight decay and a cap $\lambda_{\max}$ so that it generalizes across nearby physics rather than memorizing samples.

The policy update uses the advantage $A_r-\sum_i g_\phi(\xi)_i A_{c_i}$. In a variant, each constraint term is replaced by C3PO's ReLU penalty evaluated with domain-specific remaining budgets.

Physics settings are sampled uniformly from $\Xi$ throughout. Any effective focus on hard settings comes from the prices, not from the sampler.

The analysis has three parts. First, we will show on each benchmark that the binding constraint actually varies across $\Xi$, by plotting $\arg\max_i g_\phi(\xi)_i$ over the box. If one constraint binds everywhere, the method should reduce to a per-constraint averaged Lagrangian, and we will report that outcome as a negative result. Second, we will characterize what happens at the cap: the fraction of $\Xi$ declared infeasible, and whether shrinking the box to the feasible region recovers reward. Third, we will measure how the price network's accuracy degrades as the number of rollouts per update falls. This connects to the single-sample pacing argument.

Baselines

The naive reference is plain domain randomization with a monolithic reward, $r_{\text{task}}-\sum_i w_i c_i$, with weights tuned on the training box. It represents what most sim-to-real pipelines do. Prior evidence from Dag et al. and the SPiDR hardware gaps suggests it will fail joint feasibility. It anchors the scale of improvement.

The averaged-constraint baseline is PPO-Lagrangian from SafePO (Safety Gymnasium). It is trained under the same randomization with one constant multiplier per constraint, which is exactly our method with $\lambda_i$ forced to be constant. We also run a PID-multiplier variant and a C3PO variant with $m$ separate ReLU penalties. Beating these baselines isolates the benefit of making prices domain-dependent.

The strongest existing competitor is SPiDR (SPiDR). We extend it to $m$ costs in the most favorable way: each cost receives its own ensemble-disagreement penalty and its own coefficient, and all $m$ coefficients are tuned. Its reported figures are 100% real-world constraint satisfaction and 26.1x / 6.2x fewer violations than randomization on Go1 / RaceCar. Those figures are for a single cost. We found no reported result on Lacuna for SPiDR or any method on a multi-constraint randomized benchmark, so the comparison must be run here.

The ablations isolate the key ingredient. The first gives domain-conditioned prices to a single summed cost, testing whether the per-constraint split matters. The second uses per-constraint prices without privileged $\xi$, with prices conditioned on the actor's history embedding instead, testing whether privileged information matters.

For fairness, all methods share the same simulator, randomization box, environment-step budget, network sizes, number of seeds, and a matched hyperparameter search budget. SPiDR's per-constraint coefficients and the tightened-budget margins described below receive the same number of trials as our method's $\lambda_{\max}$ and learning rate. We count the method as beating a baseline if it achieves higher joint feasibility on held-out physics at equal or better normalized reward, with nonoverlapping 95% bootstrap intervals over seeds. A reward advantage alone at worse feasibility does not count.

Why not just…?

Why not just tighten each budget in the averaged Lagrangian, enforcing $\mathbb{E}_\xi C_i\le d_i-\epsilon_i$? This is the most dangerous alternative, because it is trivial to implement. Margins act uniformly, so a margin large enough to protect the worst corner of $\Xi$ for constraint $i$ also taxes every setting where $i$ is slack. That cost should show up as lost reward when different constraints bind in different corners. If the margins are tuned with the same search budget and they match our joint feasibility and reward, the domain-conditioned price is unnecessary and we will say so. The binding-map analysis tells us in advance on which benchmarks to expect this outcome.

Why not just run SPiDR with one variance penalty per constraint? Disagreement between dynamics models measures where the physics is uncertain, not where a given constraint is close to its budget. A state can be highly uncertain yet far from any torque limit, or nearly certain yet right at a collision boundary. SPiDR's own discussion of frozen policies under large coefficients suggests that this mismatch costs reward. The experiment compares the two at matched feasibility. If per-constraint SPiDR matches us, the contribution reduces to a benchmark and an analysis, and we will report it that way.

Why not just optimize a risk measure over domains, for example the conditional value-at-risk (the average cost over the worst $\alpha$ fraction of physics settings) per constraint? A per-constraint conditional value-at-risk objective is a fixed-shape reweighting of domains: it attends to the worst $\alpha$ fraction regardless of how infeasible those domains are. Our multipliers are the exact dual of the per-domain constraint and adapt their shape. We include this objective as an additional baseline with $\alpha$ tuned. The honest expectation is that it is close to our method when violations are concentrated in one tail, and weaker when they are spread over separate regions for separate constraints.

Why not just use an adversary that picks hard physics? RRLS found adversarial methods learn slowly because the environment distribution keeps shifting, and plain randomization beat them on average return. Our approach keeps the sampling distribution fixed and moves only the prices. We therefore do not run a full constrained adversary, and we note this omission as a limitation.

Proposed Experiments

We need benchmarks with several genuinely separate costs under randomized physics, and none exists in the file. We will build them by applying the RRLS parameter-perturbation interface (SuReLI/RRLS), which perturbs mass, friction, and damping, to Safety Gymnasium tasks.

The first benchmark is PointGoal2 in the MJX port used by SPiDR. Its costs are hazard contact, an action-energy budget, and a speed cap.

The second set is the Safety Gymnasium velocity tasks for Hopper, HalfCheetah, and Ant. Their costs are the existing velocity limit plus a cumulative squared-torque budget and a foot-slip budget. Torque should bind under heavy torso mass and slip under low friction, which is the regime the method targets.

The third is CarButton, with collision and button-misuse costs, randomized over mass and friction.

Budgets follow the C3PO convention of 25.0 for the original cost. The new costs are calibrated so that an unconstrained policy violates each one in roughly half of the box.

The primary metric is joint feasibility on a held-out grid of physics settings spanning $\Xi$. It is reported alongside normalized reward and per-constraint violation rates. A second evaluation uses a grid extending 25% beyond each side of the training box, testing behavior outside the support, a failure mode SPiDR acknowledges.

The decisive checks are three. First, on benchmarks where the binding map shows at least two constraints binding in different regions, our method must beat averaged PPO-Lagrangian, tightened margins, and per-constraint SPiDR on joint feasibility at matched reward. On benchmarks where one constraint binds everywhere, it should tie. That pattern, a win only where the binding constraint changes, is the core prediction, and it is falsified if the wins appear uniformly or not at all. Second, the summed-cost ablation must lose joint feasibility relative to the full method, confirming that keeping constraints separate is what matters. Third, the region where prices saturate must predict where held-out violations occur, judged by area under the ROC curve for the saturation flag against observed violation.

If simulation results pass these checks and hardware access permits, the natural extension is the Unitree Go1 setting from SPiDR. There, a foot-slip budget and a torque budget would be added to its safety cost, and the same comparison would be repeated zero-shot.