Robots almost never have a single goal. A reaching arm must get to its target, avoid obstacles, and move smoothly. A walking robot must track a motion, stay upright, and not shake its gearbox apart. One recent approach trains a single controller that takes a "preference weight" as input, telling it how much to care about each goal, so an operator can change the trade-off without retraining. A separate line of work, domain randomization, trains controllers across many randomly perturbed simulators (varied masses, frictions, delays) so they work on real hardware with no real-world training. The natural move is to combine the two. This note argues that the naive combination has a specific defect. A preference weight has no physical meaning, so the behavior it produces depends on the robot's actual dynamics. The same weight gives one level of jerkiness on a light, slippery robot and another on a heavy, sticky one. After transfer, the operator is back to tuning weights on hardware.
We propose a different input to the controller. Instead of dimensionless weights, the controller receives budgets in physical units, such as "keep RMS jerk below 2 m/s³ and collision-risk proxy below 0.05," and is trained to maximize task success while meeting those budgets in every randomized dynamics instance, not on average across them. Because budgets are measured quantities, one can check on the real robot whether the commanded trade-off was delivered. Because the controller has memory and sees many dynamics during training, it can learn to adjust its own internal trade-off to the robot it finds itself on. We pair this with a sampler that concentrates randomization on the dynamics where the controller most badly misses its commanded budgets. This targets exactly the "trade-off drift" that weight-conditioned policies ignore.
Core Claim
The claim has two parts. First, a policy conditioned on physical budgets and trained with per-domain constraints under dynamics randomization delivers the commanded trade-off on unseen dynamics, including real hardware, much more faithfully than a weight-conditioned policy trained under the same randomization. It does so at equal or better task success. Second, an active randomization sampler that targets budget violations, rather than generic behavioral divergence, closes the remaining misses faster than uniform or divergence-driven sampling.
Let $J_0(\pi;\xi)$ be the task return of policy $\pi$ in a simulator with physical parameters $\xi$. Let $J_k(\pi;\xi)$ for $k=1..K$ be measured cost quantities such as RMS jerk or minimum-clearance violations. Given a budget vector $b$, the policy $\pi(\cdot \mid h_t, b)$ acts on the observation-action history $h_t$ and is trained on
$$\max_\pi \ \mathbb{E}_{b,\ \xi \sim p(\xi)}\big[J_0(\pi_b;\xi)\big] \quad \text{s.t.} \quad J_k(\pi_b;\xi) \le b_k \ \text{for all } k \text{ and (with high probability) all } \xi,$$
where $p(\xi)$ is the randomization distribution. The contrast is with the weight-conditioned objective $\max_\pi \mathbb{E}_{w,\xi}[w^\top J(\pi_w;\xi)]$, whose output for a fixed $w$ is free to land at different cost levels in different $\xi$. The quantity we track is the budget-miss $\Delta(b,\xi)=\max_k \big(J_k(\pi_b;\xi)-b_k\big)_+$.
Relation to Prior Work
Several papers motivate the gap. AMOR trains one weight-conditioned policy over seven objectives in a nominal simulator. On a 20-DoF biped, operators still spent a day adjusting the smoothness weight by hand to suppress vibrations. That is direct evidence that simulated weights do not keep their meaning on hardware. We keep AMOR's idea of one policy covering many trade-offs, but we change what the trade-off is expressed in, and we train under randomization. Monolithic vs. hybrid controller shows that a scalarized reach-and-avoid policy transferred poorly to a Franka arm, and that a hand-thresholded switch between two specialist policies worked far better. Its authors flag hard-switch chatter and manual threshold tuning as open problems. Our budget input is a continuous, physically meaningful replacement for that threshold.
Other papers supply the method. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization showed on a real Fetch arm that randomizing timestep, observation noise, mass and friction, combined with an LSTM (a recurrent network), beats feedforward policies. Understanding Domain Randomization proves that memory is required for randomized training to transfer, and that non-recoverable ("non-communicating") environments make the gap grow linearly with horizon. We therefore use a recurrent policy throughout and choose tasks where recovery is possible. Active Domain Randomization supplies the sampler design: a set of sampling policies trained with Stein Variational Policy Gradient to propose simulator parameters. Its reward, however, is a discriminator's ability to tell perturbed rollouts from reference rollouts. That reward fires on benign divergence and is blind to whether a cost budget was broken. We keep the sampler machinery and replace the reward with $\Delta(b,\xi)$.
A last group of papers constrains the design. DrEureka sets randomization ranges by sweeping one parameter at a time and letting a language model choose ranges. Its safety terms are folded into a scalar reward, so it offers no trade-off control at deployment. It also warns that over-wide ranges make tasks unlearnable, which is why we pursue targeted rather than wider sampling. Distributional Pareto-Optimal MORL shows that linear weighting misses non-convex parts of a trade-off curve. Budget constraints avoid that limitation, because a constrained problem can reach points on concave regions of the curve. That paper studies return distributions within a single environment, not across randomized ones. Provably Safe Sim-to-Real Transfer gives tabular guarantees but requires real-world exploration, so we do not compete with it on theory.
Research Program
The system has three pieces. The policy is an LSTM that takes the history and a budget vector sampled per episode from a box of physically plausible values. Constraint enforcement uses an augmented Lagrangian, a standard method that adds a penalty term with learned multipliers for each violated budget. The multipliers come from a small network $\lambda(b,\xi)$ that sees the true simulator parameters during training only. This lets the effective trade-off differ across dynamics, while the deployed policy must infer those dynamics from history. The sampler is ADR's particle-based proposer, rewarded by the budget-miss $\Delta$ of the current policy.
The analysis component is empirical. For a grid of budgets, we measure how far delivered costs land from commanded costs as a function of dynamics distance from the training range. We compare this "fidelity curve" against weight-conditioned policies. We will also report the worst-domain hypervolume, meaning the size of the achieved trade-off region in the least favorable held-out dynamics. This tests whether per-domain constraints protect trade-off endpoints that average-return training loses.
Baselines
The naive reference point is AMOR-style weight conditioning in a nominal simulator with no randomization, recurrent policy, same objectives. It estimates how far trade-offs drift with no protection at all.
The strongest competitor is the same weight-conditioned recurrent policy trained under identical uniform dynamics randomization. After training, a calibration table maps weights to delivered costs in the nominal simulator, so the operator can request budgets through it. This is exactly what a careful practitioner would build from AMOR plus dynamics randomization. It shares every ingredient except the budget input and the per-domain constraints. It gets the same randomization ranges, network size, environment steps, and hyperparameter search budget (an equal number of seeds and configurations). Beating it means lower budget-miss rates on held-out dynamics and hardware, at task success no worse than its own.
On the physical Franka reach-and-avoid task, the strongest existing number is the hybrid switching controller of Monolithic vs. hybrid, which reports a 90% real success rate with a 10% collision rate. For the scalarized monolithic controller, the same paper reports 30% success with 70% collisions. We will reimplement the hybrid controller with its threshold tuned in simulation under the same randomization. We count our method as beating it if it matches 90% success at a strictly lower collision rate, with no hardware tuning, and while offering a continuous clearance budget the hybrid cannot express.
For the Fetch-style pushing task, Sim-to-Real Transfer of Robotic Control with Dynamics Randomization reports 0.89 ± 0.06 real success for its randomized LSTM. We treat that as the single-objective ceiling our budgeted policy should approach when budgets are loose.
Two ablations isolate the key ingredients. The first swaps the budget-miss sampler reward for ADR's original discriminator reward, and for uniform sampling. The second keeps budget inputs but enforces constraints only on average over $\xi$ rather than per domain, which tests whether the per-domain requirement is what buys fidelity. For ADR on its own benchmarks, no reported numeric result was found on Lacuna, only qualitative statements of lower variance than uniform sampling.
Why not just…?
Why not just train AMOR-style weight conditioning with domain randomization and memory? This is the real threat, and it is our main baseline. A recurrent policy might learn on its own to deliver consistent costs for a fixed weight, since nothing forbids it. But nothing rewards it either, because the weighted objective is indifferent to where costs land. AMOR's day of hardware weight tuning is suggestive. If the fidelity curves of the two methods coincide on held-out dynamics, our central claim is false, and we would report the contribution as limited to the sampler.
Why not just calibrate weights on the real robot with a few trials? A short real-world sweep mapping weights to delivered costs is cheap and may work if the drift is a smooth, low-dimensional shift. It fails when dynamics change during deployment, for example with a new payload or worn joints. It also costs hardware time per robot, which is what AMOR spent a day doing. We test it directly. A 20-trial real calibration of the weight baseline is compared against zero-shot budgets, both before and after attaching an unmodeled payload. We expect calibration to win on the nominal robot and lose after the change.
Why not just train one constrained policy per budget, or use the hybrid switch? Separate policies per budget are a valid upper bound on fidelity at each budget. We will train three of them as an oracle, and our single policy should come within a stated margin of them. The hybrid switch from Monolithic vs. hybrid exposes only one hand-set threshold, and its authors report chatter at the switching boundary. We measure jerk near switching events against our continuous policy.
Why not just use ADR's original sampler, or widen uniform ranges? ADR spends effort on dynamics that change behavior. Some of those changes are harmless, and it may under-sample dynamics that quietly push jerk over budget. Widening ranges risks the unlearnability DrEureka warns about. The sampler ablation measures budget-miss per unit of training compute to settle this.
Proposed Experiments
The first study is in simulation. We use MO-Gymnasium Hopper, HalfCheetah and Reacher, adding a jerk cost and randomizing mass, friction, damping, actuator gain and latency. We train on one parameter range and evaluate on progressively shifted ranges. We also include "outside-support" tests that add an effect never randomized, such as a joint backlash model. Metrics are the budget-miss rate $\Pr[\Delta>0]$, mean $\Delta$, task return, and worst-domain hypervolume over 10 seeds.
The second study uses the physical Franka reach-and-avoid task from Monolithic vs. hybrid, trained in CoppeliaSim with RealSense and AprilTag tracking. We command three clearance-and-jerk budget settings and report success, collision rate, measured minimum clearance and end-effector jerk against the commands. Each setting gets 30 trials, and the payload test is repeated.
The decisive check is simple. On held-out dynamics and on hardware, does the budgeted policy's delivered jerk and clearance stay within its commanded budgets markedly more often than the calibrated weight-conditioned policy trained under identical randomization? It must do so while matching the hybrid controller's reported 90% success. If not, the budget formulation offers nothing that weights plus memory do not already provide.
References
- AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning
- Monolithic vs. hybrid controller for multi-objective Sim-to-Real learning
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
- Understanding Domain Randomization for Sim-to-real Transfer
- Active Domain Randomization
- DrEureka: Language Model Guided Sim-To-Real Transfer
- Distributional Pareto-Optimal Multi-Objective Reinforcement Learning
- Provably Safe Sim-to-Real Transfer