Smoothing as a Locality Knob: Agent-Count-Free Imitation Through Contact

When many robots or vehicles must learn to copy expert behavior together, the cleanest training signal comes from running the learners' own policies through a differentiable simulator. A differentiable simulator is a physics engine whose outputs can be backpropagated to the policy parameters. Scoring the resulting rollouts against the demonstrations avoids covariate shift, the drift into unfamiliar states that ruins plain behavior cloning. Two things break this in multi-agent physical settings. First, contact makes the simulator's gradients discontinuous, so they are either zero or misleading. Second, in a pushing crowd or a shared rope, rigid contact carries influence across the whole group at once. Every agent's gradient then depends on every other agent, and the cost and noise of a training step grow with team size.

We propose that the same operation fixes both problems. Randomized smoothing averages gradients over small random perturbations. It turns contact cliffs into slopes, and it should also make the influence of one agent's action on another agent's future state decay geometrically with their distance in the contact graph (the graph of which bodies touch which). If that holds, each agent can be trained with a gradient truncated to its $r$-hop contact neighborhood. That neighborhood matches the local, partial observations it uses at execution. The error from truncating then shrinks as $\psi(\sigma)^r$, where $\sigma$ is the smoothing width. This would give an imitation method whose per-agent gradient cost and localization error do not depend on the number of agents. The smoothing width becomes an explicit dial trading bias in the gradient against locality.

Core Claim

Dimension-Free Rates for Natural Policy Gradient in Multi-Agent Reinforcement Learning proves that decentralized natural policy gradient converges at $O(1/\sqrt{T})$ independent of agent count $K$, with localization error $O(\psi^r)$. This requires the Dobrushin condition, meaning correlations between agents decay with graph distance. The paper assumes smooth dynamics and fixed spatial neighborhoods. Its own stated limitation is that rigid contact violates this condition. Our claim is that the violation belongs to the pointwise gradient, not to the smoothed one.

Define the smoothed influence coefficient $C_{ij}(\sigma) = \| \mathbb{E}_{w}[\partial s_j^{T} / \partial a_i^{t}] \|$. Here $s_j^T$ is agent $j$'s state at horizon $T$, $a_i^t$ is agent $i$'s action at time $t$, and the expectation is over Gaussian perturbations $w$ of width $\sigma$, as in Bundled Gradients through Contact via Randomized Smoothing. The hypothesis is

$$\sum_{j:\, d_c(i,j) > r} C_{ij}(\sigma) \;\le\; c\, \psi(\sigma)^{r}, \qquad \psi(\sigma) < 1 \text{ for } \sigma \ge \sigma_0,$$

where $d_c$ is hop distance in the time-varying contact graph and $\sigma_0$ is a threshold that depends on contact stiffness. The intuition is that a smoothed contact Jacobian is an average over configurations. In some of those configurations the contact is open and transmits nothing, so each hop attenuates the influence. If the hypothesis holds, the Dobrushin-style analysis carries over with $\psi$ replaced by $\psi(\sigma)$. The price is a bias between the smoothed and the true hard-contact objective, and that bias grows with $\sigma$.

Research Program

The method, which we call contact-local differentiable imitation, extends Imitation Learning via Differentiable Physics to many agents. That paper unrolls a single, fully observed policy through a differentiable simulator and minimizes a Chamfer-style trajectory loss. It is single-agent, fully observed, and has no convergence theory.

In our version, each agent runs a recurrent policy on its own local observation stream. Training rollouts are simulated jointly. Each agent's per-agent Chamfer loss is backpropagated only through agents within $r$ contact hops, using smoothed contact gradients. This neighborhood is recomputed from contacts as they occur, not from fixed spatial proximity. The estimator falls back to zeroth-order estimates when the first-order and zeroth-order gradients disagree, following the robust interpolation protocol of Do Differentiable Simulators Give Better Policy Gradients?. That paper works with a single agent in toy environments and asks no locality question.

Because the loss is always computed on the learners' own rollouts, no expert queries are needed. This is the DAgger-style protection against covariate shift, without the supervisor burden.

The analysis has two parts. The empirical part measures $C_{ij}(\sigma)$ against contact-hop distance across tasks and smoothing widths. The claim stands or falls on this measurement. The theoretical part attempts a bound on $\psi(\sigma)$ for penalty-based contact models. We will also check whether smoothing tames the temporal gradient explosion that Recurrent Natural Policy Gradient for POMDPs identifies for long-memory recurrent policies. That work is single-agent with smooth dynamics, and we treat its exponential wall as a constraint our method must not make worse.

We deliberately do not claim to extend theory built on gradient-domination properties of the objective, such as Shicong Cen et al. and Uri Sherman et al., to hard contact. The target is agent-count independence, not global optimality.

Testbeds

The controlled testbed is a differentiable crowd of disk robots pushing objects through rigid contact chains, built in the style of DiffTaichi. The team size $K$ is swept from 4 to 128, with demonstrations from a scripted centralized planner. Each agent observes only nearby bodies. No reported result was found on Lacuna for this testbed, so every number there will come from our own runs.

The second testbed consists of multi-gripper rope tasks adapted from DLO-Lab. Here the rope couples the grippers non-locally, which is the hardest case for the hypothesis.

The external test is the Waymo Open Sim Agents Challenge. We will write a differentiable kinematic simulator for up to 128 vehicles, with collisions and road boundaries as smoothed contacts. Partial observability there comes from each vehicle's local scene view. COMRAD is not differentiable and is not used.

Baselines

The naive reference is plain behavior cloning with the same recurrent architecture, trained to convergence on the same demonstrations. It shows how much of the gap is covariate shift, since Toward the Fundamental Limits of Imitation Learning predicts error growing quadratically with horizon.

The strongest existing method on traffic is DecompGAIL. It decomposes the adversarial realism signal, the score from a discriminator network, into ego-map and ego-neighbor terms, and trains decentralized PPO with distance-weighted neighbor rewards. It reports a Realism Meta-metric of 0.7864 on the WOSAC 2025 test split. For comparison, the WOSAC paper reports a logged oracle of 0.722 and says the best learned models then sat around 0.645. We compare against the published 0.7864 on the official split. On the disk-crowd and rope testbeds we reimplement it on the black-box version of the same simulator, with matched wall-clock and matched hyperparameter-search budgets. Beating it means matching its realism score while cutting training wall-clock substantially, or exceeding it at equal compute. Matching it only at higher compute counts as a loss.

The strongest differentiable competitor is centralized single-agent ILD applied to the joint system. It uses full backpropagation through all agents with robust-interpolation gradients and the same smoothing and tuning budget. This baseline has the most information and is the real rival at small $K$.

On the rope tasks, CMA-ES (the evolution strategy that DLO-Lab reports averaging 86.6% success, outperforming gradient methods) is run with an equal budget of simulator steps. A behavior-cloning warm-start followed by independent PPO fine-tuning covers the model-free route.

Three ablations isolate the key ingredient. The first uses truncation on the fixed spatial $r$-hop graph instead of the contact graph. The second uses smoothing with no truncation. The third uses truncation with no smoothing, meaning pointwise hard-contact gradients.

Why not just…?

Why not just warm-start with behavior cloning and fine-tune with independent PPO? On contact-rich tasks, model-free RL is very sample-hungry. PlasticineLab reports that heavily trained PPO reached normalized IoU (overlap between achieved and target shape) of about 0.80 on RollingPin, while Adam through the simulator hit 0.93 almost immediately. DLO-Lab reports PPO and SAC making no progress on Unknotting. It remains plausible, though, that PPO is competitive on the traffic benchmark, where contact is sparse. We run it at equal wall-clock there and will report it honestly if it ties.

Why not just backpropagate through the whole joint system without truncation? This is the real threat. At small $K$ the full gradient is strictly more informative, and it may win outright. Our claim concerns scaling. Truncation should hold per-agent gradient cost and variance flat as $K$ grows, while full backpropagation degrades in memory, cost, or gradient signal-to-noise ratio. The $K$ sweep settles the question. If full backpropagation matches us at $K=128$ at equal compute, the locality claim has no practical value and we will say so.

Why not just truncate by spatial distance, which is simpler? In a chain of touching bodies, two agents far apart in space can be one or two contact hops apart. A distance cutoff will then discard exactly the gradients that matter. The spatial-graph ablation tests this directly. The prediction is a gap that widens with contact-chain length and vanishes when contacts are sparse.

Why not just use DecompGAIL's local decomposition, which already localizes credit? DecompGAIL localizes the reward, not the gradient path. It still pays black-box RL sample costs and the variance of adversarial training. It also reports that smoothing the discriminator hurts, because realism boundaries are sharp. We smooth only the gradient estimator and evaluate under hard dynamics, so that failure mode should not transfer. The WOSAC comparison checks this.

Decisive Checks

The first check is the influence-decay plot. The claim fails if $C_{ij}(\sigma)$ does not decay geometrically with contact-hop distance for moderate $\sigma$. In particular, it fails if the rope tasks show flat influence at every $\sigma$ that still yields acceptable bias.

The second check is the bias–locality curve. For each $\sigma$, we measure the localization error against the achieved imitation loss under hard contact. This should reveal an interior optimum rather than a monotone trend.

The third check is the $K$ sweep on the disk crowd. Per-agent compute and final Chamfer error should be roughly constant for contact-local imitation and should grow for centralized ILD.

The fourth check is WOSAC, where we compare the realism meta-metric against DecompGAIL's 0.7864 at matched or lower training compute. A positive result on the first three checks with a loss on the fourth would still establish the scientific claim while limiting its reach. A negative result on the first check ends the project.