Actuator-Aware Fitness for Automated Reward Design in Direct-Torque Robots

Language-model reward designers such as Eureka write reward code, train policies in simulation, read back training statistics, and revise the code. This loop now beats human-written rewards on most simulated control tasks. It inherits a quiet assumption from the simulator: a motor delivers exactly the torque it is commanded. Real electric motors do not. They have cogging torque, a position-dependent pull as rotor magnets pass the stator teeth that exists even when the motor is off. Their inverters add further torque ripple. When a policy commands joint torques directly instead of going through a stiff position loop, this ripple reaches the robot's body. An automated reward search that never sees the ripple cannot learn to avoid the motions that excite it. It may also learn to rely on the ideal actuator in ways a human designer would have avoided by instinct.

The idea is to make the reward search's fitness, the ground-truth score the loop optimizes, depend on a simulated actuator whose cogging is measured from the real motors. The language model's feedback would then include the frequency content of the resulting torque ripple. It should not have to guess at generic smoothness penalties. We argue this matters for one physical reason: cogging is locked to rotor angle. It therefore falls inside the bandwidth a 200 Hz policy can affect only at slow joint speeds, which are exactly the regimes of standing, balancing, slow walking and fine manipulation. The proposal is scoped to that regime. It is also designed so that a simpler fix, such as a firmware cogging table or a better simulator with human rewards, can win outright, and the experiments are built to show whether it does.

Core Claim

Our claim is that automated reward design for direct-torque robots is misaligned at the level of its fitness function, and not only at the level of its reward terms. The mitigation that follows is fitness evaluation against an identified, position-periodic actuator model, with ripple spectra fed back into the language model's revisions. We predict this yields policies with lower hardware vibration at the cogging order and no loss in task performance, compared with three alternatives: automated rewards evaluated on ideal actuators, hand-tuned smoothness terms, and actuator-augmented simulation paired with a fixed reward.

One relation sets where the policy can matter at all. The cogging frequency seen at a joint is

$$f_{cog} = \frac{\mathrm{LCM}(Q, P)\, N\, |\dot q|}{2\pi},$$

where $Q$ is the number of stator slots, $P$ the number of magnet poles, $\mathrm{LCM}$ their least common multiple (the number of cogging cycles per rotor revolution), $N$ the gear ratio, and $\dot q$ the joint speed in rad/s. A policy running at 200 Hz can only shape content below 100 Hz, by the sampling limit. Its influence is therefore confined to joint speeds under roughly $2\pi \cdot 100 / (\mathrm{LCM}(Q,P) N)$. Above that speed, and for inverter dead-time harmonics, the policy cannot act as a cancelling controller. Those effects belong to the motor firmware.

This bandwidth limit is a deliberate departure from the research file's suggestion that reward design could discover "counter-harmonic injection." The harmonic cancellation that Influence of harmonic voltage coupling on torque ripple of permanent magnet synchronous motor reports happens at the electrical level, far above any policy's bandwidth. We do not claim it.

Where This Sits

This work builds directly on the automated-design line and supplies the piece it lacks. Eureka writes and mutates reward code using an explicit fitness score. It reports beating human rewards on 83% of 29 tasks, with a 52% average normalized improvement. DrEureka adds safety-oriented torque penalties and has the language model set domain-randomization ranges for sim-to-real transfer. It reports 34% higher forward velocity and 20% greater distance than human-tuned rewards on a real Go1 quadruped. Both methods evaluate fitness on Isaac Gym's ideal joints. DrEureka's torque penalty is a sum of squared torques, and that term has no notion of rotor angle. We keep their search loop and replace what the fitness is measured on.

The motivation comes from direct-torque learning. Learning Torque Control for Quadrupedal Locomotion shows that a 200 Hz torque policy on the A1 recovers from kicks that defeat position control. DecAP makes torque policies cheap to train, about 25 minutes in Isaac Gym, by adding decaying priors from a PID controller that tracks a position-based teacher. Both use hand-weighted reward terms, and both simulate the motors as ideal torque sources. They establish that the direct-torque regime is worth having. They do not tell us whether cogging matters in it. We found no reported result on Lacuna for cogging-induced ripple or vibration in A1 or Go1 learned policies, so establishing that effect size is our first job.

The method for the actuator model comes from Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. That work identified a DC-motor model and a 15–19 ms latency on Minitaur, reaching 1.18 m/s on hardware against 1.34 m/s in simulation. Its model is linear and has no position-periodic term. We extend the system identification with a cogging profile measured per motor.

The motor-control literature both constrains and supports the approach. The iterative learning control of Iterative Learning-Based Negative Effect Compensation Control of Disturbance stores cogging as a table indexed by position, cutting pointing error by 54–70% relative to active disturbance rejection. That result is evidence that cogging is repeatable enough to identify, and it is also a warning that firmware compensation may be enough on its own. The observers of Analysis and Compensation Control of Dead-Time Effect on Space Vector PWM remove the 6th-order dead-time current harmonic, reported as about 37.5 mA at 100 rpm, inside the drive. We therefore leave dead-time effects to firmware and do not ask the policy to handle them.

Research Program

We would build three components, each on the same robot.

The first is a per-motor cogging model. We identify it by rotating each joint slowly under a stiff external position loop, with the load removed, and recording the torque or current needed at each rotor angle. We fit a Fourier series in rotor angle whose fundamental order is $\mathrm{LCM}(Q,P)$. Held-out slow sweeps validate the fit. Reflected through the gearbox, the model is added to Isaac Gym's joint torque as $\tau_{applied} = \tau_{cmd} + \tau_{cog}(N q)$, together with the latency model of Tan et al.

The second is an actuator-aware fitness. Each candidate policy is scored in two ways. One score is task success, taken from the benchmark's original success measure. The other is the spectral energy of body acceleration and of $\tau_{cog}$-driven joint velocity ripple in a band around $f_{cog}$, measured over evaluation rollouts. To avoid bringing back human weighting at the meta level, which is the known weakness of Eureka-style loops, the loop keeps a Pareto set over these two scores and never collapses them into a weighted sum. Final policies are compared across the whole front.

The third is spectral reflection. Eureka's text feedback, which is per-term training statistics, is extended with a short summary: the dominant ripple frequencies, the joint speeds at which they occur, and which reward terms correlate with them. The language model receives the cogging model's parameters, not the electromagnetic equations. This keeps the context short, which addresses the file's concern that full motor physics would crowd out task reasoning.

Tasks are chosen to sit in the low-speed regime where the policy has influence: quadruped standing balance under pushes, slow command-tracking walking at 0.2–0.5 m/s, and a slow single-leg foot-placement task. On the real robot, we measure IMU vibration spectra, joint encoder ripple, velocity-tracking error and electrical energy per meter.

Baselines

The naive reference point is DrEureka run unchanged on ideal actuators, as in DrEureka, and deployed on hardware with stock firmware. This is where the field is today. DrEureka's reported Go1 gains of 34% velocity and 20% distance come from faster, higher-speed locomotion than our tasks. We would therefore re-run it on our low-speed tasks rather than borrow those numbers.

The strongest hand-designed competitor is the human-weighted torque reward of Learning Torque Control for Quadrupedal Locomotion. We augment it with the second-order action-smoothness penalty of Grad-CAPS, which reduced SAC Walker action fluctuation from 0.53 to 0.41 without losing return. This combination is the best existing way to buy smoothness without an actuator model.

The strongest competitor overall is DrEureka trained in our cogging-augmented simulator with its fitness unchanged, deployed with a firmware anticogging table built from the same identification data. This baseline has every physical ingredient we have. It lacks only the actuator-aware fitness and the spectral reflection.

All automated methods get the same language model, the same number of generations and samples per generation, the same PPO budget and the same seeds. The human-reward baselines get a weight-tuning budget equal in GPU hours to the automated search.

Beating these baselines means two things. At matched task success, measured by velocity-tracking error or push-recovery rate within the baseline's seed confidence interval, our method must show a statistically clear reduction in hardware IMU energy in the cogging band. And it must show that reduction against the strongest baseline, not only against the naive one.

The key ablation removes the spectral reflection but keeps the Pareto fitness. This separates "the fitness sees the ripple" from "the language model is told about the ripple." A second ablation runs our method with the anticogging table switched off, to measure how much the policy compensates on its own.

Why not just…?

Why not just add the cogging model to the simulator and keep a fixed reward? This is the most serious threat, and it is why the strongest baseline exists. Tan et al.'s success with an identified actuator model on Minitaur suggests that better simulation alone closes much of the sim-to-real gap. A policy trained against $\tau_{cog}$ may learn to avoid slow dwell near high-cogging angles without any reward change. If the strongest baseline matches our hardware vibration at equal task success, the conclusion is that the simulator was the bottleneck, not the reward designer. We would report that as the result.

Why not just compensate cogging in motor firmware with a position-indexed table? Cogging is repeatable, and the iterative learning result above shows table-based compensation works well on a single axis. The honest expectation is that firmware removes most of the motor-level ripple. What it cannot remove is gearbox-reflected residue, temperature drift in magnet strength, and the policy's own tendency to hold slow, low-torque configurations where residual ripple dominates. The ablation with firmware on versus off, crossed with fitness type, measures exactly how much is left for the policy. If the residual is below IMU noise, the reward-side contribution is not worth its cost.

Why not just use action-smoothness penalties like Grad-CAPS? Those penalties act on the policy's output. Cogging is a function of rotor angle, not of how the action changes over time. A perfectly smooth torque command at slow joint speed still produces ripple at $f_{cog}$. Conversely, smoothness weights strong enough to suppress it in a proxy sense tend to over-smooth; the Grad-CAPS paper reports this for first-order penalties. The hand-designed baseline tests this directly. It would win only if the ripple we measure turns out to come mostly from policy jitter rather than from cogging, which the vibration spectra will reveal.

Why not just go back to position control? Stiff PD loops do absorb ripple. But the kick-recovery results of Chen et al. and the rough-terrain transfer of DecAP show what direct torque buys. We include a position-control policy as a reference on the push-recovery task. If it matches direct-torque policies on vibration and push recovery in our low-speed regime, the motivating tension largely disappears, and we would report that.

Proposed Experiments

The first decisive check is whether the problem exists. We deploy DrEureka and human-reward torque policies on the Go1 and compare IMU and encoder spectra against the predicted $f_{cog}$ across joint speeds. We need a visible peak that tracks joint speed as the formula predicts, and that disappears above the speed where the cogging frequency passes 100 Hz. If no such peak rises above structural and gait vibration, the project stops here, and the negative result is itself worth reporting given that no Lacuna-indexed work has measured it.

The second check is whether automated rewards exploit the ideal actuator more than human rewards do. We compare the gap between simulated and real ripple energy for the two policy families. A larger gap for automated rewards would mean the search has overfit the ideal motor.

The third is the full cross of fitness type (task-only versus actuator-aware Pareto), reflection (on versus off), simulator (ideal versus cogging-augmented) and firmware table (on versus off). This design runs on the three low-speed tasks with five seeds each and reports Pareto fronts of task success against real vibration.

The claim is supported only if actuator-aware fitness moves the hardware front outward relative to cogging-augmented simulation with a fixed fitness plus firmware compensation. If the fronts coincide, the finding is that better simulation plus better firmware is enough, and that reward-search misalignment with actuators is not a practical concern at these speeds. That answer would also be useful for deciding where automated reward design should and should not spend its effort.