When a team of robots trained in simulation is moved onto real hardware, most of what goes wrong sits in fast, low-level physics: motors that are weaker than modeled, extra latency, friction, backlash. Almost none of it sits in the slow, strategic layer that decides who passes to whom. Reinforcement learning has a cheap, well-studied knob for making a learner distrust its own model of the world: the discount factor, which sets how far into the future the learner looks. Shortening that horizon suppresses compounding model error. The catch is that it also makes agents myopic, and myopia breaks team coordination. We propose to avoid that trade-off by splitting the control system into a slow team-strategy loop and a fast per-robot reflex loop. The slow loop keeps a long horizon so coordination survives. The fast loop gets a short horizon, measured in seconds and set separately for each robot according to how badly that robot's simulated actuation disagrees with its real behavior.
In two sentences for a colleague: discount-factor regularization is a known remedy for model error, but applied to a whole multi-agent learner it destroys cooperation. In a two-rate hierarchy you can apply it only to the fast per-robot loop, where the sim-to-real error actually lives, and calibrate it per robot, which lets you get the robustness without the coordination cost.
Core Claim
Three lines of theory motivate using the horizon as a regularizer. Biasing Approximate Dynamic Programming with a Lower Discount Factor shows that with an imperfect function approximator, the best discount for the solver is strictly lower than the task's true discount. The Dependence of Effective Planning Horizon on Model Accuracy shows that the discount controls the size of the policy class and produces a U-shaped error curve when planning with an inaccurate model. On the Role of Discount Factor in Offline Reinforcement Learning shows, for linear models, that lowering the discount is equivalent to planning against the worst dynamics inside a confidence set.
These results give us the mechanism: a short horizon acts as pessimism against dynamics you do not trust. They also constrain us. The equivalences are proved in tabular, linear, or offline settings, so whether they hold for deep policies trained online under domain randomization is an empirical question this proposal must answer, not assume.
The tension we resolve is stated plainly in the literature. FACMAC shows that continuous multi-agent teams need centralized, long-horizon credit assignment to escape "relative overgeneralisation," a failure where each agent's independent improvement locks the team into a poor joint behavior. A globally shortened horizon pushes exactly toward that greedy regime.
Our claim is that the two effects live at different timescales and can be separated. We express every horizon in physical seconds $\tau$ rather than in steps. A loop that acts every $\Delta t$ seconds then uses
$$\gamma = \exp(-\Delta t / \tau),$$
so the same $\tau$ means the same look-ahead whether the loop runs at 25 Hz or 500 Hz. The slow centralized loop keeps a long $\tau_H$ chosen for the task. Each robot $i$'s fast reflex loop uses its own short horizon
$$\tau_i = c / \varepsilon_i,$$
where $\varepsilon_i$ is that robot's one-step actuation-model error measured on a small batch of target-domain data, and $c$ is a single constant tuned once. The inverse form is a hypothesis: roughly, a robot whose per-step model error is twice as large should be trusted for half as long, if error compounds about linearly over short windows. The experiments test whether one $c$ transfers across tasks.
The falsifiable predictions are specific. Held-out-dynamics team performance should be U-shaped in $\tau_{\text{fast}}$ and flat or increasing in $\tau_H$. Shortening $\tau_H$ should hurt coordination-heavy tasks even when it helps robustness. Per-robot $\tau_i$ should beat a shared fast horizon only when hardware wear is heterogeneous.
Relation to Closest Work
Policy Gradient for LQR with Domain Randomization is the closest use of the discount in sim-to-real. It anneals the discount upward during domain-randomized policy-gradient training to find a controller that stabilizes all sampled systems, and reports 90% real success on a rotary pendulum. There the discount is a temporary optimization aid for one linear system, removed by the end of training. Here it is a permanent, per-agent, per-rate regularizer in a nonlinear multi-agent hierarchy.
Two papers combine hierarchy with transfer but treat the discount as a fixed default. EvoControl pairs a 31.25 Hz PPO planner with a 500 Hz evolution-strategies reflex on a single arm. VolleyBots uses a strategy/motion hierarchy for drone teams, but transfers only a single-drone drill to hardware. Neither asks how the horizon should differ across rates or across robots.
Two papers establish the multi-agent reality-gap problem and motivate the per-robot design. Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning finds that actuation noise, not sensing noise, collapses shared team learning. Joint-Local Grounded Action Transformation finds that simultaneous per-agent grounding updates destabilize neighbors. Our method corrects nothing in the simulator, so it cannot trigger that cascade; instead it changes how much each agent trusts its own fast dynamics.
Research Program
The architecture has two levels. The slow level runs at 20–30 Hz under centralized training with decentralized execution: during training a shared critic sees the global state and all agents' actions, while at run time each agent acts on local observations only. We implement this level with MAPPO and, in a second arm, with FACMAC's factored critic. It outputs per-robot setpoints.
The fast level runs a per-robot reflex at 200–500 Hz. It is initialized from a proportional-derivative (PD) controller and annealed into a learned network, following EvoControl's controller annealing. The fast reward is setpoint tracking plus effort, discounted with each robot's own $\gamma_i = \exp(-\Delta t_L/\tau_i)$.
This design removes the gradient conflict the file flags between evolution-strategies reflexes and FACMAC's centralized gradients. The centralized gradient never passes through the fast loop, and the slow critic treats the fast loop as part of the environment. We run both PPO and evolution strategies at the fast level, because a short physical-time horizon may make gradient methods viable at high rates. That is a secondary finding worth reporting either way.
We estimate $\varepsilon_i$ from about a minute of target-domain rollouts per robot under the initial PD controller. It is the normalized one-step prediction error of the simulator's actuation model against observed joint or rotor responses. Training uses domain randomization in MuJoCo Playground and Isaac Sim, so that the sweeps over $\tau_{\text{fast}}$, $\tau_H$, and $c$ fit a modest compute budget.
The analytic component is a measurement study rather than a new theorem. For each task we will trace real or held-out return against $\tau_{\text{fast}}$ and against $\tau_H$ separately. We will check whether the best $\tau_{\text{fast}}$ tracks $1/\varepsilon$ across robots with different injected wear. We will also measure how much fast-loop pessimism shifts the dynamics the slow critic sees, which is the hierarchical non-stationarity problem.
Baselines
The naive reference point is a flat, single-rate MAPPO team at 50 Hz with $\gamma = 0.99$ and the same domain randomization. We give it the same simulator steps and a hyperparameter budget equal to the proposal's.
The strongest existing system for the multi-agent hierarchical setting is a VolleyBots-style hierarchy with fixed default discounts at both levels. On 3v3 volleyball it reports a 69.5% win rate against self-play (VolleyBots). We rerun it with our randomization and our held-out dynamics so that it differs from our method only in the horizon choices. Beating it means a higher win rate against the same fixed opponent pool under held-out drone dynamics, without losing win rate under nominal dynamics.
For single-robot reflex quality, the reference is EvoControl. It reports a normalized Ant return of 368 against 100 for the best flat baseline (EvoControl). We reproduce its fast loop inside our hierarchy as the "fixed-discount reflex" arm.
The most dangerous baseline is the best single global discount. We sweep $\gamma$, applied identically at both rates and to all agents, with the same number of trials the proposal spends tuning $c$. On the Role of Discount Factor in Offline Reinforcement Learning reports that simply moving $\gamma$ from 0.99 to 0.95 rescued SAC on noised D4RL. If a single tuned global value matches us, the rate split is not worth its complexity.
We also include a discount-annealing arm in the style of Policy Gradient for LQR with Domain Randomization, and an ensemble-pessimism arm in which the fast critic's value is penalized by the disagreement of five dynamics-randomized critics.
Three ablations isolate the key ingredients. The first shortens $\tau_H$ instead of $\tau_{\text{fast}}$, which tests the coordination-cost prediction. The second uses a shared $\tau_{\text{fast}}$ instead of per-robot $\tau_i$, which isolates the calibration. The third holds $\gamma$ fixed per step rather than per second while changing control rate, which tests whether physical-time horizons are the right unit.
For MACPO's safe multi-agent MuJoCo suite we found no reported result on Lacuna for per-environment returns, so there we compare only against our own reruns using the released code (MACPO code).
Why not just…?
Why not just sweep one global discount? This is a real threat, and the global sweep baseline exists to settle it. Our argument that it should fall short rests on the coordination tension documented by FACMAC: a global value low enough to absorb actuation error also shortens the team's credit assignment. The decisive evidence is the $\tau_H$ ablation. If shortening the slow horizon costs nothing on the passing and volleyball tasks, the tension is not real at these timescales, and a global sweep would be the honest recommendation.
Why not just randomize the simulator more broadly? Broad randomization already underlies every arm. The file notes that it tends to produce passive policies that underperform under nominal conditions. It also cannot express that robot 3 is worn while robot 1 is not, unless randomization is made per-robot and the policy is told which robot it is. We include a per-robot-widened-randomization arm, scaled by the same $\varepsilon_i$, as a direct competitor. If it matches per-robot $\tau_i$ on held-out dynamics and nominal return, the discount adds nothing beyond what the randomization width already provides.
Why not just penalize value uncertainty with an ensemble? Ensemble pessimism targets the same failure more locally and could plausibly work as well. Its disadvantages are compute and the need to tune a penalty scale. The discount instead comes, per Discount Factor as a Regularizer in Reinforcement Learning, with a state-distribution-aware penalty on value magnitude at no extra cost. The ensemble arm, run with matched tuning trials, decides this. We will report wall-clock cost alongside return, so a tie counts as a result.
Why not just keep a fixed PD tracker as the low level? Then there is no fast learner to regularize, and the fast loop cannot compensate for wear at all. EvoControl reports that fixed PD tracking lacked the compliant reactions its learned reflex provided on a real arm. We keep a fixed-PD hierarchy as an arm. If it matches our method on held-out dynamics, the right conclusion is that the fast loop should not be learned for these tasks.
Proposed Experiments
The first testbed is TIERS/NoisyKukaReacher from Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning, extended to two rates. The original reports a mean score near −2 under sensing noise, collapsing to about −6 when 15 or more of 30 arms carry variable actuation noise. We reproduce the OV15 and OV25 conditions. The decisive check is whether per-robot $\tau_i$ keeps the team near the sensing-noise level of −2 where the flat baseline falls to −6, and whether the shared-horizon ablation lands in between.
The second testbed is multi-agent MuJoCo Ant and ManyAgent Swimmer, trained in MuJoCo Playground and evaluated on held-out actuator gain, latency, and friction with heterogeneous per-agent wear. Here we trace the two horizon curves and test whether one $c$ tuned on Ant transfers to Swimmer.
The third testbed is VolleyBots 3v3 under held-out rotor constants and latency. It carries the coordination test, because passing and setting require long-horizon credit.
For hardware, we will first check the fast-loop rule on a single robot. On the Quanser Qube Servo 2 the domain-randomized policy-gradient method's 90% real success is the number to match or exceed. If a Franka arm is available, we will also repeat EvoControl's reactive-contact task on it. Multi-robot hardware results, such as two drones passing or two arms reaching, are a stretch goal. Our primary claims rest on sim-to-sim transfer to held-out dynamics, which we state openly as a limitation.
The metrics are held-out return or win rate, nominal-dynamics return (to detect over-conservatism), constraint-violation cost on the safe suite, and fast-loop tracking error. We run five seeds per arm and report confidence intervals.
The proposal succeeds if three things hold. Rate-split, per-robot horizons must beat the best global discount on held-out dynamics while matching it under nominal dynamics. Shortening $\tau_H$ must measurably hurt the coordination tasks. And the best fast horizon must track $1/\varepsilon_i$ across robots.
References
- Biasing Approximate Dynamic Programming with a Lower Discount Factor
- The Dependence of Effective Planning Horizon on Model Accuracy.
- On the Role of Discount Factor in Offline Reinforcement Learning
- FACMAC: Factored Multi-Agent Centralised Policy Gradients
- Policy Gradient for LQR with Domain Randomization
- EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
- VolleyBots: A Testbed for Multi-Drone Volleyball Game Combining Motion Control and Strategic Play
- Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning
- Joint-Local Grounded Action Transformation for Sim-to-Real Transfer in Multi-Agent Traffic Control
- MuJoCo Playground
- Discount Factor as a Regularizer in Reinforcement Learning