Adversarial Motion Priors That Remember: Memory-Matched Discriminators for Delayed Physical Systems

One way to avoid hand-writing a reward function for a legged robot is to learn it. In this approach, called an adversarial motion prior, a classifier is trained to tell the robot's motion apart from reference motion such as dog motion capture, and the robot is rewarded for fooling it. This replaces dozens of hand-tuned penalty terms with a single learned "style" score. The classifier, known as the discriminator, only ever looks at one step of motion at a time: the current pose and the next pose. That is enough when the next pose depends only on the current one. Real bodies do not behave this way. Actuators lag their commands, series-elastic and viscoelastic joints store and release energy over hundreds of milliseconds, and muscles and batteries fatigue. When the observed state forgets things the physics remembers, a one-step discriminator can be fooled by motion whose step-to-step statistics look right while its longer-range timing is wrong.

We propose to give the discriminator a small, physically structured memory instead of a large generic one. The memory is a running weighted integral of past states. Its weighting kernel is built from a few learned delays and decay rates, or from a power law, and can be updated in constant time per step. The bet is that this structure matters most where adversarial imitation is actually used: tiny reference datasets, such as the roughly 4.5-second German Shepherd clip in Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions. With that little data, a recurrent or attention-based discriminator has enough capacity to memorize the clip, while a few-parameter delay kernel does not.

Core Claim

Matching one-step transition statistics recovers the reference trajectory distribution only if the reference process is Markov in the observed state. When it is not, the adversarial reward has a blind spot. Two motion processes can share every pairwise transition statistic and still differ in three-step or longer statistics, and a transition discriminator gives both the same score. We claim three things. First, this blind spot is measurable in simulated locomotion with injected actuator delay and viscoelastic joints. Second, a discriminator that also receives a memory feature $m_t$ closes most of the gap. Third, when $m_t$ comes from a low-parameter delay or decay kernel, it closes the gap with less reference data and less discriminator overfitting than a recurrent or transformer discriminator of matched quality.

The memory feature is

$$m_t = \sum_{j=1}^{J} c_j \int_{0}^{t-\tau_j} e^{-\lambda_j (t-\tau_j-s)}\, \phi(x(s))\, ds,$$

where $x(s)$ is the observed robot state at time $s$ and $\phi$ is a small learned feature map. Each of the $J$ components has its own learned delay $\tau_j$, decay rate $\lambda_j$ and weight $c_j$. Because each term is a delayed exponential, $m_t$ updates recursively from its previous value and a buffer of length $\max_j \tau_j$. This avoids the growing cost of evaluating a general integral kernel. A sum of exponentials can also approximate the power-law kernels of fractional dynamics. The style reward is then $r^S_t = 1 - \tfrac14\,(D(x_t, x_{t+1}, m_t) - 1)^2$, the least-squares form used by AMP, with the same gradient penalty.

Relation to Prior Work

Two papers motivate the gap and supply the base method. AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control introduced the transition-level discriminator reward. Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions showed that on a quadruped it beats a complex hand-designed reward on energy efficiency and transfers to hardware. Both score $(s_t, s_{t+1})$ pairs or very short windows with an MLP. We keep their reward form, regularization and task reward, and change only what the discriminator can see.

The memory mechanism draws on the continuous-time history models. Neural Delay Differential Equations showed that delayed-state dynamics can separate trajectories that pass through the same point but have different pasts, which an ODE cannot do. This is exactly the ambiguity a one-step discriminator suffers from. Neural Integro-Differential Equations split dynamics into a local part and a non-local integral part, which is the split between $(x_t, x_{t+1})$ and $m_t$ that we impose. Non-Markovian Reinforcement Learning using Fractional Dynamics showed in glucose control that a single fractional exponent can carry long memory cheaply. That supports our use of few-parameter kernels over large hidden states. All three use history kernels to forecast or control dynamics. None uses them to define a learned reward, which is our contribution.

The work on non-Markovian rewards constrains and sharpens the claim. Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning uses a causal transformer to turn delayed composite returns into step rewards. Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning does the same with an LSTM that sees both history and the current transition. Both learn rewards from return labels, and the history-dependence they model sits in the reward rather than in the physics. Our setting is the reverse: the reward is learned from demonstrations without labels, and the history-dependence comes from the dynamics. Their architectures become our strongest generic-memory competitors. The automata line, Reinforcement Learning with Non-Markovian Rewards and Adaptive Reward Design for Reinforcement Learning, needs discrete event alphabets and does not apply to continuous gait timing. We do not build on it.

Research Program

The first piece is a diagnostic. We build families of reference processes that share their pairwise transition statistics but differ in longer statistics, such as gaits with identical per-step joint kinematics but different lag between commanded and realized foot placement. We then check whether a trained transition discriminator separates them. This turns the blind spot from an argument into a measurement. The same tool gives a cheap test of whether a given reference dataset is far from Markov, by comparing a one-step discriminator with a history discriminator on held-out reference windows. We then report how the method's gains scale with the size of that gap.

The second piece is the discriminator itself: a recursive delayed-exponential memory, with $J$ between one and eight, fed with the current transition into the standard AMP MLP. Delays are learned through a soft interpolation over the buffer so gradients reach $\tau_j$. We also train a fractional variant whose kernel is a fixed exponential-sum fit to a power law with learned exponent. Training follows AMP's PPO setup without changes. The policy in every condition, including baselines, gets the same recurrent memory. This matters because a non-Markov environment also handicaps a memoryless policy, and we want to isolate the effect of the reward.

The third piece is an analysis of what the learned delays recover. In simulation we know the injected actuator delay and joint relaxation time, so we can check whether the learned $\tau_j$ and $\lambda_j$ land near them. If they do, the discriminator is interpretable in a way a GRU's hidden state is not.

Baselines

The naive reference point is a hand-designed multi-term locomotion reward of the kind Escontrela et al. compare against, tuned with the same number of tuning runs as every learned method. The strongest existing method is AMP with its standard transition discriminator, as in Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions. That paper reports a Cost of Transport of 0.93 to 1.12 for AMP versus 1.37 to 1.65 for complex hand-designed rewards on its quadruped. That result was obtained without injected delays, and we found no reported result on Lacuna for AMP under actuator delay or viscoelastic joints. We therefore reproduce AMP in our delayed environments rather than borrow its numbers.

Three generic-memory discriminators are the real competitors. The first is AMP over a stacked window of $k$ frames, with $k$ tuned. The second is a GRU discriminator, following the history-plus-transition design of the CSC-LSTM in Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning. The third is a small causal transformer discriminator in the style of the backbone in Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning. We also run a τ-GRU discriminator following Gated Recurrent Neural Networks with Weighted Time-Delay Feedback, since it is the closest recurrent model with explicit delay. All discriminators get the same reference data, gradient penalty, PPO budget, policy architecture and random-search budget over their own hyperparameters. Parameter counts are reported alongside results.

Two ablations isolate the key ingredient. The first fixes a single delay at the true injected value, which shows whether learning the kernel matters. The second gives the memory to the policy but not the discriminator, which shows whether the gain comes from the reward. An oracle discriminator that receives the privileged hidden simulator state, such as actuator internal state and spring deflection, sets the upper bound. The proposal beats AMP if it closes at least half of the long-horizon distribution gap between AMP and this oracle at equal task return and Cost of Transport. It beats the generic-memory discriminators if it reaches their matching quality with reference data cut to a few seconds, where we expect them to overfit.

Why not just…?

Why not just stack more frames into the discriminator? For short, fixed, integer delays this may work as well as our method, and we treat it as a serious competitor rather than a straw man. It degrades in two cases. When the memory is long or spread out, as with viscoelastic relaxation or fatigue, the window must grow until the discriminator input becomes large. When the window is large, a 4.5-second clip gives few independent windows to train on. The sweep over delay length and relaxation time will show where stacking stops keeping up. If it never stops in our ranges, we will report that the structured kernel is unnecessary for those regimes.

Why not just use a GRU or transformer discriminator? Generic memory can in principle represent any kernel. The evidence that structure helps is indirect. On noise-padded CIFAR-10, a delay-structured recurrent model reached 62.7% against 11.6% for an LSTM (Gated Recurrent Neural Networks with Weighted Time-Delay Feedback), which shows that plain recurrence struggles to carry long gradients. Fractional models also controlled glucose well from small data (Non-Markovian Reinforcement Learning using Fractional Dynamics). Neither result is about discriminators. The reference-data-size sweep is the decisive test: if GRU and transformer discriminators match ours at every data size, the structured memory adds nothing beyond interpretability.

Why not just append recent actions to the discriminator input? When the only hidden memory is pure command delay, the recent action history makes the state Markov again. This alternative may then fully solve the problem, and we include it explicitly. It cannot recover latent mechanical memory such as spring deflection, tissue relaxation or fatigue, which is driven by states rather than recent commands. We separate the two cases with environments that have only actuator delay and environments that have only viscoelastic joints.

Why not just retune the hand-designed penalties? The comparison already exists without delays, with AMP beating hand-designed rewards on Cost of Transport (Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions). Under delay, the hand-designed reward would also need terms that depend on timing, which reintroduces the engineering burden we are trying to remove. We keep it as the naive point of comparison.

Proposed Experiments

The testbed is MuJoCo locomotion (HalfCheetah, Ant, Walker2d) and an AMP-style quadruped, modified in three controlled ways. The first adds a first-order actuator lag with a time constant between 0 and 200 ms. The second adds series-viscoelastic joints with a relaxation time in the same range. The third adds a slow fatigue state that scales available torque. In each modified environment, an expert policy trained with privileged state on the true task produces reference trajectories, so the ground-truth reference distribution is known. We vary reference data from about 4.5 seconds, matching the German Shepherd clip, up to several minutes. We also run the real retargeted dog clip in the delayed quadruped, but only for qualitative and Cost of Transport comparison, since its ground-truth memory is unknown.

The metrics are task return, Cost of Transport, and long-horizon distribution match between policy and reference. Distribution match is measured by maximum mean discrepancy over one-second windows, the error in the joint-velocity autocorrelation spectrum, and the lag between commanded and realized foot contacts. We also measure the discriminator's gap between training and held-out reference windows, which is our measure of overfitting, and how close the learned delays $\tau_j$ come to the injected ones.

Three outcomes would decide the question. If the transition discriminator does not lose distribution match as delay grows, the blind spot does not matter in practice and the project stops at the diagnostic. If action-history or frame-stacked discriminators match ours across all delay types, the structured kernel is unnecessary. If our kernel matches the GRU and transformer discriminators only at large reference sizes and beats them at small sizes, while recovering delays near the injected values, the core claim holds.