Self-Tuning Networked Filters for Sim-to-Real Robot Swarms

A robot trained in simulation sees clean sensor readings. On hardware, the same robot sees noise whose size and correlation structure nobody measured beforehand. Every robot in a team faces this, and each robot's estimate of where it is also depends on what its neighbors report. We propose to keep the control policy trained in simulation fixed. On hardware, each robot instead retunes a small local filter, the recursive rule that blends its motion prediction with incoming measurements into a state estimate. The only training signal is how badly the filter predicts the next sensor reading, a quantity the robot can compute on its own without any ground truth. This works through estimation–control duality, the classical fact that designing an optimal filter is mathematically the same problem as designing an optimal controller run backward in time. Recent work used that fact to turn filter tuning under unknown noise into a well-behaved gradient-descent problem, but only for a single centralized system.

The question is whether this guarantee survives in a team. When every robot's filter gain is restricted to use only its neighbors' information, and all robots update their gains at once, the dual problem becomes a decentralized control design problem. Such problems usually lose the nice optimization landscape. Our central bet is narrower and testable. For teams of interchangeable robots, the symmetry of the team breaks the dual problem into a few small, independent single-robot problems. The landscape stays benign, and the amount of real-world data needed does not grow with team size. Where that symmetry fails, we will measure how badly things degrade.

Core Claim

Data-driven Optimal Filtering for Linear Systems with Unknown Noise Covariances and Duality-Based Stochastic Policy Optimization for Estimation with Unknown Noise Covariances write steady-state Kalman gain learning as gradient descent on the prediction-error cost

$$J(L) = \mathbb{E}\Big[\tfrac{1}{T}\textstyle\sum_{t=1}^{T} \lVert y_t - H\hat{x}_t(L)\rVert^2\Big],$$

where $y_t$ is the raw measurement, $H$ the known measurement map, $L$ the filter gain, and $\hat{x}_t(L)$ the filter's predicted state. They show this cost is coercive: it blows up as the filter approaches instability. It also satisfies a gradient-dominance (Polyak–Łojasiewicz) property, so stochastic gradient descent converges globally and linearly. Policy optimization of finite-horizon Kalman filter with unknown noise covariance proves the finite-horizon analogue. None of this work treats multiple agents.

We extend the setting to $N$ agents with identical local dynamics, coupled through a graph with adjacency matrix $W$. We restrict the gain to the neighbor-local form

$$L = I_N \otimes L_{\text{self}} + W \otimes L_{\text{nbr}},$$

where $L_{\text{self}}$ and $L_{\text{nbr}}$ are small $d\times d$ blocks shared by all agents and $\otimes$ is the Kronecker product. We claim three things. First, when $W$ is the mean-field (all-to-all, scaled by $1/N$) coupling, or more generally commutes with the dynamics coupling, the dual problem block-diagonalizes in the eigenbasis of $W$. It splits into a "team mean" mode and a "deviation" mode, each a $d$-dimensional dual LQR problem. Gradient dominance then follows from the single-agent result, and the deviation-mode gradient averages over $N-1$ statistically identical copies. The number of real trajectories needed for a fixed accuracy should therefore be flat or decreasing in $N$, rather than growing as it would for a centralized $Nd$-dimensional filter. Second, for sparse graphs the shared gains couple the eigenmodes, so gradient dominance is not guaranteed. We will characterize empirically, and where possible analytically, how the landscape degrades with graph spectral spread. Third, the filter is retuned from the robot's own prediction errors, so the step needs no motion capture, no ground-truth state, and no model of the real noise. This makes it a direct sim-to-real tool for the sensor-noise gap.

The contrast with the closest multi-agent work matters. Attention-Based Recurrence for Multi-Agent Reinforcement Learning under Stochastic Partial Observability (AERIAL) shows that centralized critics trained on the true simulator state fail under sensor noise, and replaces the state with attention over recurrent memories. It learns this filtering implicitly and without guarantees, and it cannot retune itself on hardware without reward. Mean-field methods such as Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learning get population-independent rates from exchangeability, but they assume perfect observations. We borrow their symmetry argument and apply it to the estimation side rather than the value function.

Research Program

The theory step formalizes the networked dual problem. In it, each agent's filter is a structured gain, the noise covariances are unknown, and the dual is a structured LQR over the adjoint of the coupled dynamics. We will prove the block-diagonal decomposition and the resulting convergence and sample-complexity statement for commuting couplings. We will then quantify a perturbation bound in the non-commuting case that depends on how far $W$ is from commuting with the dynamics coupling. We will also address concurrent updates directly. Joint-Local Grounded Action Transformation for Sim-to-Real Transfer in Multi-Agent Traffic Control documents a "cascading invalidation" effect, in which simultaneous local updates destabilize neighbors' models. That paper is a constraint on our design. Under shared gains and synchronous gradient steps, our updates are steps on a single joint objective rather than a game between agents, so cascading invalidation should not arise. If agents use independent, asynchronous gains it may, and we will test both regimes. We inherit two limitations from the single-agent work and state them openly. The dynamics matrices must be known, which the simulator supplies here: the gap we target is in the sensors, not the dynamics. The theory also needs many short independent trajectories, which on robots means repeated short episodes with resets.

The algorithm is deliberately plain. We train the policy in simulation on filter estimates rather than true states, which is the AERIAL lesson applied with an explicit filter. On deployment, we freeze the policy and run decentralized stochastic gradient descent on $(L_{\text{self}}, L_{\text{nbr}})$. Each robot computes its local gradient from its own prediction errors and neighbor messages, and the team averages the gradients with one round of consensus per episode. For nonlinear robots we apply the same procedure to the linearized error dynamics of an extended filter. In that setting the guarantees become heuristics, and we say so.

The testbeds are three. The first is a synthetic linear networked swarm (double-integrator agents on ring, grid, and all-to-all graphs), scaled from 4 to 1,000 agents, where optimal gains are computable for reference. The second is the physical Duckietown setup of Transferring Multi-Agent Reinforcement Learning Policies for Autonomous Driving using Sim-to-Real, modified so the robots localize from onboard wheel odometry and camera lane features. The 120 Hz OptiTrack motion capture that paper fed to its policies becomes an evaluation-only ground truth. The third is the CityFlow-to-SUMO traffic grid of the JL-GAT paper, with injected loop-detector noise in SUMO as a nonlinear, sim-to-sim check. MessySMAC from the AERIAL paper serves as a stress test outside our assumptions, because its sign-flip noise is not additive Gaussian.

Baselines

The naive reference point is the simulation-trained policy deployed unchanged, either on raw noisy observations or with a filter whose noise covariances were simply copied from the simulator. Every method must beat this, and the gap between it and the motion-capture-fed policy defines the "sensor gap" we try to close. We will report the fraction of that gap each method recovers.

The strongest estimation baseline is a centralized dual policy-optimization filter: the algorithm of Data-driven Optimal Filtering for Linear Systems with Unknown Noise Covariances run on the full $Nd$-dimensional joint system. It is the best possible version of our objective without the locality constraint, so it bounds what locality costs. It gets the same real trajectories and the same step-size tuning budget. We count beating it as matching its steady-state prediction error within a few percent while using fewer trajectories and far less communication as $N$ grows. The file reports its convergence guarantees ($T=O(\ln(1/\epsilon))$, $M=O(1/\epsilon)$) but no numerical error on a shared benchmark; no reported result found on Lacuna for this baseline on a multi-agent task.

The strongest closed-loop baseline is MAPPO with medium domain randomization, as in the Duckietown paper. That paper reports real-world reward 1.85 times that of a Gipps rule-based controller, but with motion-capture state. We retrain it with sensor-noise randomization added and the same three-hour single-GPU simulation budget the paper reports. Beating it means higher real-world reward under onboard sensing, without the conservative "paranoid" driving that paper attributes to high randomization, which we will measure as mean speed. The second closed-loop baseline is AERIAL. The file reports only that it outperformed QMIX and QPLEX on MessySMAC, with near-zero return versus about −15 for state-based methods on Dec-Tiger; no reported per-map MessySMAC win rate found on Lacuna. We run it on the same simulation budget. On traffic, the reference is JL-GAT's reported 4x4 rainy average travel time of 467.11 s (down from 485.63 s). We report travel time with and without detector noise for JL-GAT and for JL-GAT plus our filter.

Two ablations isolate the key ingredient. The first optimizes the same prediction-error loss with an unconstrained per-agent gain and no neighbor term, which isolates locality and coupling. The second uses a GRU state estimator trained on the same prediction-error loss on hardware, which isolates the duality-derived linear structure from the self-supervised signal itself.

Why not just…?

Why not just estimate the noise covariances on hardware and run a standard Kalman filter per robot? This is the most serious threat. Classical innovation-based covariance estimation followed by a per-robot filter could work well when coupling between robots is weak. Our argument against it is specific. With neighbor-coupled dynamics, the best neighbor-local gain is generally not the Kalman gain of any per-robot $(Q,R)$ pair, so per-robot identification optimizes the wrong target. Classical adaptive schemes also lack the non-asymptotic guarantees the dual formulation provides. The synthetic swarm settles this. We sweep the coupling strength and report the regime where the classical approach matches ours; if it matches everywhere, our contribution shrinks to the guarantee, and we will say so.

Why not just randomize sensor noise in simulation and train a recurrent policy? The Duckietown study found a Goldilocks effect, where too much randomization produces frozen, overcautious robots. Randomization also has to guess the real noise distribution in advance, while our method measures its consequences after deployment. Still, a well-chosen randomization level might suffice. The MAPPO and AERIAL comparisons under onboard sensing test this directly, and we will report the best randomization level found under an equal tuning budget rather than a weak one.

Why not just fine-tune a neural filter on hardware with the same prediction-error loss? This keeps our signal and drops our structure. It is exactly the GRU ablation. We expect it to need far more real trajectories and to scale poorly with $N$, because it has no symmetry-induced decomposition. If it matches our sample efficiency, the duality machinery buys only interpretability and stability, and we will report that outcome.

Decisive Experiments

Three results decide the proposal. The first is a scaling plot on the synthetic swarm: real trajectories needed to reach 5% excess prediction error, versus $N$ from 4 to 1,000, for all-to-all and ring graphs. The claim fails if the all-to-all curve grows with $N$. The second is a landscape test under concurrent updates: linear convergence rate versus graph spectral spread, with shared synchronous gains compared against independent asynchronous gains. This tells us whether multi-agent non-stationarity destroys gradient dominance, the unresolved tension between the single-agent theory and JL-GAT's cascading invalidation. The third is the physical Duckietown run. We measure position error against the evaluation-only motion capture and task reward under onboard sensing, report the fraction of the motion-capture-to-onboard gap recovered after a fixed number of retuning episodes, and compare against randomized MAPPO, AERIAL, per-robot adaptive filters, and the GRU ablation. MessySMAC is reported as an honest out-of-assumption check, where we expect the advantage over AERIAL to narrow or vanish.