Telling Apart "I Got Worse" from "They Got Better": WoLF-Paced Action Blending for Continuous Games

When a reinforcement learning agent's payoff drops, two different rules from two literatures give opposite advice. Single-agent "cautious" methods say to slow down: the drop probably means your last update was bad, so change less. The multi-agent "Win or Learn Fast" (WoLF) rule says to speed up: in a game, a drop usually means your opponent found a way to exploit you, so move away quickly. Win or Learn Fast means an agent learns slowly while it is ahead of its historical average and quickly while it is behind. Both rules are right in their own regime. Neither can tell which regime it is in, because both look at the same scalar signal: did my return go down?

We propose to separate the two causes with a counterfactual comparison that becomes cheap under one specific policy-update scheme, deterministic policy interpolation. In this scheme, the executed action is a weighted average of the old policy's action and a newly proposed policy's action, with a single mixing weight $\beta$. Because the old and new policies define a one-dimensional path in action space, we can score both endpoints against the same current opponent with a centralized critic. If the new policy is worse than the old one against that same opponent, the agent made itself worse, and we brake. If both are worse than the agent's historical baseline, the opponent moved, and we accelerate. The result is a single, smooth rule for setting $\beta$ in continuous-action games. It reduces to cautious interpolation when the opponent is fixed and to WoLF when the agent's own update is sound.

Core Claim

Our claim is that a continuous-action agent in a competitive game learns more stably than either cautious interpolation or classic WoLF alone if it does three things. First, it blends actions deterministically rather than sampling between policies. Second, it sets the blending weight from two separate signals: its own-update quality, measured against a fixed opponent, and opponent-driven loss, measured against a slowly tracked baseline. Third, it brakes on the first signal and accelerates on the second. "More stable" here means lower exploitability at the last iterate, less oscillation, and smoother control signals.

Concretely, let $\mu_k$ be the current deterministic actor, $\nu_k$ a candidate obtained by a few gradient steps, $\sigma_t$ the opponent's current behavior, and $\hat J(\cdot;\sigma_t)$ a critic estimate of return against that behavior. Let $\bar J_t$ be an exponentially decaying average of past returns, in the style of the tracking baselines of Schmid et al.. The executed policy is $\pi_\beta(s) = (1-\beta)\mu_k(s) + \beta\nu_k(s)$, with

$$\beta_k = \mathbb{1}\big[d^{\text{own}}_k > 0\big]\left(\beta_{\min} + (\beta_{\max}-\beta_{\min})\,\mathrm{sigmoid}\!\big(-d^{\text{opp}}_k/\tau\big)\right) + \mathbb{1}\big[d^{\text{own}}_k \le 0\big]\,\beta_{\text{brake}},$$

where $d^{\text{own}}_k = \hat J(\nu_k;\sigma_t) - \hat J(\mu_k;\sigma_t)$ isolates the effect of the agent's own update and $d^{\text{opp}}_k = \hat J(\mu_k;\sigma_t) - \bar J_t$ isolates the effect of the opponent's drift. The temperature $\tau$ replaces WoLF's hard switch between two learning rates, which the file notes as a source of discontinuity and tuning burden. The two indicator terms are exhaustive: the agent either brakes to $\beta_{\text{brake}}$ or sets $\beta$ by the WoLF sigmoid.

Relation to Prior Work

The mechanism for smooth action blending comes from Smooth Imitation Learning for Online Sequence Prediction. That work introduced deterministic interpolation for continuous control and argued, via Jensen's inequality, that it performs at least as well as stochastic mixing while avoiding jitter. It used a fixed mixing schedule in a single-agent imitation setting.

The braking half of our rule comes from two single-agent methods. Cautious Actor-Critic shrinks its interpolation weight when a moving-average advantage signal predicts degradation. Cautious policy programming grows its weight with expected advantage. Both treat any drop as self-inflicted, which in a game is exactly the wrong response to being exploited. We keep their brake but gate it on the counterfactual $d^{\text{own}}$ rather than on raw return.

The accelerating half comes from WoLF, from Multi-Agent Systems and its continuous extension GraWoLF in Simultaneous Adversarial Multi-Robot Learning. GraWoLF varies a gradient learning rate over a stochastic Gibbs policy. It performs no interpolation, and it reports sensitivity to how the baseline is initialized.

Two multi-agent works come closest in spirit. Multi-Agent Trust Region Learning also forms a mixture of current and trial policies, but it chooses the mixture by solving a small meta-game at every step, using sampled opponent policies. Two-Timescale Q-Learning with Function Approximation in Zero-Sum Stochastic Games uses a convex interpolation step with last-iterate guarantees, but its step size follows a fixed decaying schedule that ignores whether the player is winning. Our contribution is the performance-sensitive, cause-separated schedule, not interpolation itself.

One constraint must be stated up front. GraWoLF's learned attacker was genuinely mixed: its most probable action occurred only about 40% of the time. A deterministic actor cannot represent a mixed equilibrium, and the Jensen argument needs concavity in actions, which a best-responding opponent breaks. We therefore scope the claim to games whose equilibria are pure or nearly pure, such as saddle-point pursuit and blocking games, and we test the failure case explicitly rather than hiding it.

Research Program

The work has three parts.

The first is the algorithm. We build a deterministic actor-critic in which each agent keeps an anchor actor and a candidate actor, and each agent's centralized critic conditions on its own action and the opponent's action, as in MADDPG. Each agent uses its critic to evaluate $d^{\text{own}}$ and $d^{\text{opp}}$ on a recent batch of opponent states and actions, then distills the blended policy back into a single anchor. Because the two endpoints lie close together in action space, the critic is queried only slightly off its training distribution. We measure this shift, since it is the main threat to the counterfactual signal. To address the baseline-initialization bias reported for GraWoLF, the baseline $\bar J$ uses a bias-corrected exponential average, initialized from the first measured return rather than from a constant.

The second part is analysis in linear-quadratic and concave-convex continuous games. In these games action interpolation reduces to a variable-step gradient-play dynamic whose equilibrium is known, so we can ask whether the asymmetric schedule gives last-iterate convergence where a fixed step cycles. We will attempt to extend the Moreau-envelope Lyapunov argument of the two-timescale paper to steps that stay within a bounded ratio of a decaying envelope. This is a stated attempt, not a promised result.

The third part is an empirical study across competitive continuous benchmarks, with a single-agent sanity check. The sanity check tests one prediction of the design: with a fixed opponent, $d^{\text{opp}}$ is uninformative, so the rule should behave like cautious interpolation.

Baselines

The naive reference point is independent deterministic actor-critic with plain gradient updates and no interpolation. This is DDPG run per agent, which Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments used as its comparison. MADDPG itself, with a centralized critic and no pacing, is the strongest widely used method for continuous competitive tasks in the file. MADDPG shares our critic architecture, so it isolates what the schedule adds.

For the general-sum setting the strongest existing competitor is MATRL Multi-Agent Trust Region Learning. It is the only method in the file that explicitly builds stabilizing mixtures of current and trial policies in multi-agent games. It reported removing cycling in two-agent Atari Pong and a total reward of 18 versus −2 for independent learners on the MA-Gym Checker game. For our main competitive continuous tasks, we found no reported result on Lacuna for MATRL exploitability or win rate, and none for MPE Predator-Prey or the attacker-defender task beyond qualitative statements. We will therefore rerun MATRL ourselves.

GraWoLF, reimplemented on the same deep critic, represents WoLF without interpolation. Cautious Actor-Critic transplanted to each agent represents interpolation with caution only. The two-timescale scheme with its fixed decaying $\beta_k$ represents interpolation with a time-based schedule.

Three ablations isolate the key ingredient. The first uses WoLF-only pacing on raw return, which drops $d^{\text{own}}$. The second uses caution-only pacing, which drops $d^{\text{opp}}$. The third applies the same two-signal rule to stochastic mixing instead of action averaging.

All methods share network sizes, the critic, environment steps, and seeds (at least ten per cell). Each gets the same hyperparameter search budget, tuned on a held-out game instance. Beating the field means lower final-iterate exploitability than MATRL and MADDPG at matched steps, with confidence intervals that do not overlap, and without higher wall-clock cost than MATRL's reported 1.1 to 1.2 times that of independent learners.

Why not just…?

Why not just put WoLF on the optimizer learning rate, as GraWoLF does? This is the most serious threat, because it is simpler and already demonstrated on physical robots. The argument for action-space blending is that a small parameter step in a deep network can still move actions a lot in some states, whereas blending bounds the per-state action change by $\beta$ times the gap between the two policies. That argument is untested. The GraWoLF-on-deep-critic baseline, with the same two-signal gating, settles it directly. If it matches our method on exploitability and action smoothness, we will report that the contribution is the cause-separated signal and not interpolation.

Why not just use Cautious Actor-Critic in each agent? In a game, caution alone treats exploitation as noise and holds the agent in place while it is being exploited. We predict that caution-only agents will show long plateaus of low return against an adapting opponent. The caution-only ablation, and its time spent below baseline, measures this directly.

Why not just line-search $\beta$ greedily against the current opponent? The critic makes this cheap. However, it amounts to a damped best response, and best responses cycle in self-play, which is exactly the problem WoLF was designed to prevent in Multi-Agent Systems. We include greedy line search as a baseline and expect it to win against fixed opponents but oscillate in self-play.

Why not just use MATRL? MATRL is a strong competitor. Its meta-game step, however, needs sampled opponent policies and a small equilibrium solve every update, whereas our rule needs two critic evaluations and a running average. If MATRL wins on exploitability, the remaining question is whether our method is competitive at lower cost, and we will report both.

Proposed Experiments

We start with linear-quadratic two-player games and a continuous blocking game with a known pure saddle point, where we can measure the exact distance to equilibrium at the last iterate. We then add a continuous matching-pennies-style game whose only equilibrium is mixed. There we expect every deterministic method, ours included, to fail, and we report it as a negative control.

Next come the competitive MPE tasks from the MADDPG paper, Predator-Prey and Physical Deception, and a simulated reconstruction of the attacker-defender task from GraWoLF. On these we measure four things. Approximate exploitability is the return a freshly trained best-response agent achieves against the frozen final policy. Oscillation is the amplitude of return over the last fifth of training. Action jerk is the mean squared change in action between consecutive steps. Maximum policy degradation is the largest drop in return between successive iterations.

Two decisive checks close the study. In single-agent Ant-v2 with no opponent, our method should roughly match Cautious Actor-Critic's reported maximum degradation of 1811, against 7793 for Soft Actor-Critic Cautious Actor-Critic. A large gap in either direction would mean the counterfactual signal is misfiring. In self-play, logging which branch of the rule fires should show acceleration concentrated right after opponent policy changes and braking right after bad candidate updates. If the two branches fire at random relative to these events, the core claim fails regardless of final scores.