Hand-Designed Rewards as Integrals Over Physical Time

People who hand-write rewards for robot control usually write them "per step": a bonus of 1 for staying upright, a penalty on squared torque, a term for forward velocity. Nothing in that reward records how long a step lasts. When the simulator's control frequency changes, the per-step reward silently changes its meaning, and every weight a practitioner tuned becomes wrong. Common examples are raising the action-repeat, switching integrators, or moving from simulation to a robot running at another rate. This note proposes treating each hand-designed reward term as what it physically is. A term is either a rate that accumulates over time, which should be integrated over the step with a numerical quadrature rule, or an event that fires once, which should not be. The claim is that a large share of what gets reported as reward-weight and learning-rate sensitivity in continuous control is a discretization artifact. Rewriting rewards in physical time, and integrating them with a quadrature rule matched to the simulator's substeps, should measurably lower that sensitivity.

The argument has two parts. First, the time units of reward terms can be made explicit and consistent. Second, once they are, we ask whether the order of the quadrature rule matters, which is the high-order integration question. Hyperparameter sensitivity is the lens throughout. Success is not a higher peak return. Success is a smaller penalty for using one fixed set of weights and learning rates across many tasks and control frequencies.

Core Claim

Write the hand-designed reward as a sum of rate terms $\rho_i(s,a)$ with weights $w_i$ plus event terms $e_j$ with weights $u_j$. The per-step reward that respects physical time over a control interval of length $\Delta t$ is

$$ r_t = \sum_i w_i \int_0^{\Delta t} e^{-\beta \tau}\, \rho_i(s(t+\tau), a_t)\, d\tau \;+\; \sum_j u_j\, e_j(s_t, a_t, s_{t+1}), $$

where $\beta$ is a discount rate in inverse seconds, so that the per-step discount is $\gamma = e^{-\beta \Delta t}$. Standard practice drops the $\Delta t$ and evaluates $\rho_i$ once, at the end of the step.

The consequence is sharpest for the constant "alive" bonus that Hopper, Walker2d and Ant use. Exploit Reward Shifting in Value-Based Deep-RL shows that adding a constant $b$ to every reward is equivalent to shifting the value initialization by $b/(1-\gamma)$. Suppose one adopts the discount rescaling $\gamma(\Delta t)=\gamma^{\Delta t}$ recommended by Making Deep Q-learning methods robust to time discretization but keeps the per-step bonus $b$. The effective offset is then about $b/(\beta\Delta t)$, which blows up as the control step shrinks. If the bonus is treated as a rate, the offset stays at $b/\beta$ independent of $\Delta t$.

This makes a falsifiable prediction. Sensitivity reductions should be largest on alive-bonus environments and smallest on Swimmer and HalfCheetah, which have no alive bonus. Changing $\Delta t$ also shifts the ratio between rate terms and event terms. No reward normalization can undo that shift, because normalization rescales the whole reward and leaves the ratio unchanged.

Relation to Prior Work

Three papers motivate the proposal by exposing discretization effects in RL. Each stops short of reward specification. Making Deep Q-learning methods robust to time discretization proves that Q-learning degenerates as the step shrinks. It derives scaling laws for the learning rate and discount, and fixes the value parameterization with Deep Advantage Updating (DAU). It treats the reward as given and assumes Euler-style transitions. An Idiosyncrasy of Time-discretization in Reinforcement Learning shows that the standard return mixes a left-point Riemann sum for discounting with a right-point sum for rewards. It proposes a first-order correction (the right-point, or RP, return), tests it with REINFORCE on Servo Reacher, and explicitly leaves higher-order quadrature open. Continuous-time Model-based Reinforcement Learning integrates rewards and dynamics with adaptive high-order ODE solvers. It does so only for smooth, analytically known rewards in systems of four or fewer dimensions, and never measures tuning sensitivity.

We differ from all three in the same way. We operate on the hand-written reward itself, distinguishing rate terms from event terms. We apply quadrature to rewards sampled at the simulator's existing internal substeps, so no learned model or differentiation through a solver is needed. And we evaluate by sensitivity rather than by peak return.

Two lines of work supply the method. The metrics are the hyperparameter sensitivity $\Phi$ and the effective dimensionality $d$ of A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning. $\Phi$ is the performance lost by using one best fixed configuration everywhere. $d$ is the number of hyperparameters that must be tuned per environment to reach 95% of fully tuned performance. The quadrature rules come from numerical analysis. The Average Vector Field construction in Energy-preserving Runge-Kutta methods shows how the choice of quadrature rule (trapezoidal, Simpson, Gauss) determines which properties a discretized line integral preserves.

One paper constrains the design. Adkins et al. report a "normalization paradox": per-minibatch advantage normalization raises peak return but increases learning-rate sensitivity, while percentile scaling lands in the "mirage" region where gains depend on tuning. Any fix we propose must therefore be judged on the performance-sensitivity plane, not on return alone.

Research Program

Simulator. We build on Brax. Its environments already advance physics through several internal substeps per control step, and its accelerator throughput makes Adkins-style sweeps of hundreds of thousands of runs affordable.

Reward decomposition. We annotate every term of the standard reward for Ant, HalfCheetah, Hopper, Walker2d and Swimmer as either a rate or an event. Examples of rates are forward velocity, the alive bonus and torque cost. Termination and contact-onset penalties are events.

Discretizations. We implement four ways of computing each step's reward. The standard per-step reward is the reference. The unit-consistent right-point reward multiplies each rate by $\Delta t$ and evaluates it at the end of the step. The trapezoidal reward averages the start and end values. Simpson and two-point Gauss-Legendre rules evaluate each rate at the internal substeps, applying the $e^{-\beta\tau}$ weight inside the rule.

Scaling. All variants use Tallec et al.'s $\gamma^{\Delta t}$ discount and learning-rate scaling. This isolates the reward specification from the already-known fixes.

Environment set. The set for computing $\Phi$ and $d$ is the product of task and control step, with action-repeat of 1, 2, 4 and 8 times the default. The hyperparameter grid covers learning rate, entropy coefficient, and the reward weights on the velocity, alive and control terms. The algorithms are PPO and SAC.

Theory. Alongside the experiments, we will derive the bias of each quadrature rule in the one-step reward integral for smooth reward rates. The right-point rule's error grows as $O(\Delta t^2)$ per step, while trapezoidal and Simpson errors grow as $O(\Delta t^3)$ and $O(\Delta t^5)$, from standard Taylor remainders. We will then show how this bias enters the value target through the geometric sum over the horizon.

We are also honest about a threat. Brax rewards involving contact are not smooth within a step, and there the high-order advantage may vanish. This is the regime flagged by Yildiz et al. and by DAU's weaker HalfCheetah results. We will report it separately rather than average it away.

Baselines

The naive reference point is the environment's default per-step reward with the default discount and learning rate, trained at each control step with no adjustment. It shows how much sensitivity practitioners currently absorb. Its $\Phi$ and $d$ are computed on exactly the same grid, seeds and step budget as every other condition.

The strongest existing competitor for the time-discretization part of the problem is the combination of Tallec et al.'s discount and learning-rate scaling with the right-point return of the Idiosyncrasy paper. For off-policy SAC we also add a DAU-style value parameterization. This baseline represents everything the literature already knows about discretization short of touching the reward. We run it with the identical grid, seeds and compute. The file contains no reported result for $\Phi$ or $d$ for this combination on Brax, nor any reported number for RP returns beyond the qualitative claim that it "never performed worse" than the standard return on Servo Reacher. We therefore must establish it ourselves. Beating it means a lower $\Phi$ at equal or better fully tuned return, which places our method in Region 1 of the performance-sensitivity plane relative to this baseline.

The strongest competitor for the sensitivity part is per-setting automated tuning. ARLO reports that its automated pipeline found DDPG settings scoring above 4600 on HalfCheetah-v3. It also reports that tuned SAC reached a return of -8.2 on a Linear Quadratic Gaussian task against an optimum of -7.2, compared with -59.0 for default settings. In our design, per-setting tuning is exactly the "fully tuned" reference against which $\Phi$ is measured. A fair comparison therefore asks two questions. How close does our single fixed configuration get to per-setting tuned performance? And how many tuned hyperparameters ($d$) does each condition need? We count cost in environment steps consumed by tuning.

We also run a normalization baseline: running reward normalization plus per-minibatch advantage normalization under the standard reward. This is the most common practical remedy for scale problems.

The key ablation separates the two ingredients. Unit-consistent right-point integration tests whether time units alone explain the gain. Moving from right-point to trapezoidal to Simpson/Gauss at a fixed unit convention tests whether quadrature order adds anything beyond that.

Why not just…?

Why not just shrink the time step so discretization error disappears? Tallec et al. prove that value-based methods lose the ability to rank actions as the step goes to zero, so a smaller step alone makes learning worse, not better. More to the point, control frequency is often dictated by hardware or by action-repeat choices rather than chosen freely. Our sweep over action-repeat directly measures what happens when the step changes. The naive baseline at the smallest step will show whether small steps fix anything on their own.

Why not just let AutoRL find the right weights for each frequency? It can, and in our design it defines the ceiling. ARLO itself reports hundreds of compute hours for pipeline search, and Adkins et al. argue that gains obtained through per-environment search are a mirage when tuning cannot be repeated. This alternative is a real threat only if tuning is cheap in the target setting. The experiment settles it by reporting $d$ and tuning cost. If our method leaves $\Phi$ unchanged, AutoRL wins and we say so.

Why not just normalize rewards or advantages? Normalization rescales the whole reward, but changing $\Delta t$ alters the ratio between rate terms and event terms, which no global rescaling can restore. Adkins et al. also found that advantage normalization raises learning-rate sensitivity. We treat this alternative as a genuine competitor and include it as a baseline. If it matches our $\Phi$ on the alive-bonus environments, the unit-consistency claim is wrong.

Why not just use the right-point return fix, since higher-order quadrature may be overkill? This is the most plausible threat to the high-order component. The first-order correction may capture nearly all of the benefit. The ablation over quadrature order answers it directly, and we expect order to matter mainly at large action-repeat, where rate terms vary substantially within a step. If order never matters, the contribution shrinks to unit-consistent reward specification. That is still a useful and simple result, and we will report it in that form.

Decisive Checks

The central test is the performance-sensitivity plane on the task-by-control-step environment set, for PPO and SAC. The unit-consistent reward should lower $\Phi$ relative to the Tallec-plus-RP baseline without lowering tuned return. The reduction should be concentrated on Hopper, Walker2d and Ant, where the alive bonus dominates, and near zero on Swimmer.

The second test is weight transfer. We tune reward weights at the default step, deploy them unchanged at 4 and 8 times the action-repeat, and measure the retained fraction of tuned return. Unit-consistent rewards should retain markedly more.

The third test concerns quadrature order. Gains from trapezoidal and Simpson/Gauss rules over the right-point rule should grow with action-repeat and should disappear on reward terms dominated by contact, where smoothness fails. Observing that pattern, and not a uniform improvement, is what would show that integration order is doing real work rather than acting as a disguised change of reward scale.