Consensus in a Learned Linear Space for Partially Observed Agent Teams

Agents in a team usually see only a small part of the world, so each must guess the rest. They can share guesses with neighbors, but there is a catch. In a nonlinear world, averaging two agents' estimates does not produce a meaningful third estimate, because averages of valid states need not be valid states. Classical distributed estimation avoids this problem only by assuming linear dynamics, where averaging is principled and convergence can be proved. We propose to learn a coordinate system in which the team's shared world evolves linearly, and to run a classical neighbor-to-neighbor averaging filter in those coordinates.

Each agent encodes what it sees into this shared space, predicts forward with one learned linear operator, and averages with its neighbors. The decoded estimate feeds a fully decentralized policy. Centralization is used only to train the encoder against the true global state. It is not used in a value function shared across agents, and it is not needed at execution. Two properties follow. Per-agent computation and communication depend on the size of the learned space and on the number of neighbors, not on team size. And the stability of the team's estimation error becomes a linear-algebra question that can be checked and regularized. Learned multi-agent state inference has neither property today.

Core Claim

Partial observability in cooperative multi-agent reinforcement learning (MARL, learning policies for several interacting agents) is currently handled by one of two kinds of model. The first is a black-box recurrent or graph model, as in SIDE, which reconstructs a latent state with a variational graph auto-encoder. The second is a generative model, as in GlobeDiff, which samples plausible global states with a conditional diffusion model (a generator that turns noise into samples by repeated denoising). Both work well. Both pool information with learned nonlinear operations that offer no guarantees, and both behave unpredictably when messages are lost, when the communication graph changes, or when more agents are present at test time than in training.

Our claim is that a Koopman-style lifting can make the classical "consensus plus innovation" estimator of Fast and Robust State Estimation and Tracking via Hierarchical Learning applicable to nonlinear multi-agent tasks. A Koopman lifting maps a nonlinear state into a higher-dimensional space where its evolution is approximately linear. In the consensus-plus-innovation estimator, each agent mixes its neighbors' estimates with a correction from its own new measurement. That estimator's convergence depends on subnetwork diameter rather than global diameter, and it tolerates dropped packets, but only under linear models. We expect the lifted estimator to match generative state inference on standard partially observed benchmarks. We expect it to degrade far more gracefully under message loss and changes in team size. Where the hidden state is truly multimodal, we expect it to lose to diffusion, and we will measure where that boundary lies rather than assume it away.

Mechanism

Let $z_t \in \mathbb{R}^d$ be the lifted team state, with $d$ fixed and independent of team size. The lifted state evolves as $z_{t+1} \approx K z_t$ under a learned matrix $K$.

Agent $i$ applies a learned encoder $\phi$ to its local observation $o^i_t$ and its own action. The output, $y^i_t = \phi(o^i_t)$, is treated as a linear measurement $y^i_t \approx C_i z_t$, where $C_i$ is selected by the agent's type or role. Agent $i$ then updates its estimate $\hat z^i_t$ as

$$\hat z^i_{t+1} = K\Big(\hat z^i_t + \sum_{j \in \mathcal{N}_i} w_{ij}(\hat z^j_t - \hat z^i_t)\Big) + L_i\big(y^i_{t+1} - C_i K \hat z^i_t\big).$$

Here $\mathcal{N}_i$ is the set of neighbors whose messages arrived at step $t$, $w_{ij}$ are mixing weights, and $L_i$ is a learned correction gain. Agent $i$ transmits only $\hat z^i_t$, which is $d$ numbers.

A permutation-invariant decoder maps $\hat z^i_t$ to task-relevant aggregates rather than to the full state. Examples are enemy health and position summaries per region, or teammate role summaries. This follows the reasoning of Scale-Free Estimation of the Average State in Large-Scale Systems, which shows that a dimension-one observer can track an average with error that does not grow with network size when controllers need only aggregates.

If the lifting residual is bounded by $\delta$, the stacked error across agents evolves as $e_{t+1} = M(G)\,e_t + \text{(terms of order } \delta)$. The matrix $M(G)$ depends on $K$, the gains $L_i$, the measurement maps $C_i$, and the communication graph $G$. Estimation error therefore stays bounded whenever the spectral radius $\rho(M(G)) < 1$. We penalize this spectral radius during training over randomly sampled graphs and packet-drop masks.

The certificate holds only relative to the learned linear model. It says the filter's error does not diverge on its own, and that error grows linearly with how badly the lifting fits. This is weaker than the plant-level guarantees of Plug-and-play distributed state estimation for linear systems. It is much more than SIDE or GlobeDiff provide.

Relation to Prior Work

Three papers supply the method. The consensus update and its robustness to dropped links come from Fast and Robust State Estimation and Tracking via Hierarchical Learning. The idea of each node tracking only what it needs, without copying the whole global state, comes from Distributed Observers for LTI Systems. Both are restricted to linear time-invariant plants with known matrices. We replace known matrices with learned lifted ones.

Data-driven parallel Koopman subsystem modeling and distributed moving horizon state estimation for large-scale nonlinear processes is the closest existing work. It fits per-subsystem Koopman models with hand-chosen lifting functions and solves distributed moving horizon estimation (an optimization over a sliding window of recent measurements) on chemical plants. It does not learn liftings from raw observations, does not handle agents that act, and does not let the estimate feed policy learning.

Two lines of work motivate the proposal. The first is the multi-agent state-inference line: SIDE, GlobeDiff, and the teammate-awareness model LINDA, which predicts Gaussian distributions over teammates' trajectories. These show that explicit state inference matters. The second is COMRAD, which reports that centralized critics in methods such as MAPPO and HAPPO can hurt role-asymmetric agents, to the point that independent PPO wins. This motivates putting centralization into estimator training rather than into the value function.

One paper constrains the design. GlobeDiff argues that discriminative estimators blur or collapse when one local history fits several global states. Our filter's posterior is effectively unimodal per agent, so this argument directly challenges us. Our bet is that pooling neighbors' measurements reduces the ambiguity that a single agent faces. That bet is testable, and it may fail on some maps.

Research Program

The work proceeds in three stages.

Estimation in isolation. We train $\phi$, $K$, $C_i$, $L_i$ and the decoder with supervised losses against logged global states, collected under a fixed behavior policy. The losses cover multi-step prediction in the lifted space, decoding error of the aggregates, and the spectral-radius penalty. This lets us measure estimation quality separately from reinforcement learning.

Joint training with decentralized policies. We couple the estimator with policies trained by independent PPO, each conditioned on the agent's local history plus its decoded estimate. Training alternates between fitting the estimator on fresh on-policy data and updating the policies. A CTDE variant, with a centralized critic and the same estimator, is also run. Comparing the two directly tests whether moving centralization from the critic to the estimator helps.

Deployment stress tests. We vary three conditions: message-drop rates from 0 to 50%, random rewiring of the communication graph, and test-time team sizes up to twice the training size. The latter is possible because $d$ is fixed and the decoder pools over agents.

Baselines

Naive references. The do-nothing reference is recurrent MAPPO or independent PPO with no explicit state inference. On the GlobeDiff partially observed variant of SMAC v1 (sight range 3, enemy types and health masked, map 6h vs 8z), GlobeDiff reports a win rate of about 0.12 or lower for vanilla MAPPO. A second reference is a communication-free version of our estimator (all $w_{ij}=0$), which shows how much consensus contributes at all.

Strongest existing method. The strongest competitor is GlobeDiff, which reports a win rate of about 0.47 on that same 6h vs 8z setting. It is the strongest because it targets exactly the one-to-many ambiguity of partial observability, and it does so with a model more expressive than ours. As a second state-inference baseline we run SIDE, which reports a 71% median win ratio on SMAC 27m_vs_30m against 16% for VDN.

Fair comparison. For fairness, GlobeDiff is given the same neighbor information at execution time: its conditioning input is extended with neighbors' encoded observations, compressed to the same $d$ numbers per neighbor per step. All methods share the same environment-step budget, the same hyperparameter-search budget (an equal number of trials per method), at least five seeds, and the same policy-learning backbone. Beating GlobeDiff means one of two outcomes:

  • matching its win rate within seed confidence intervals with no message loss, while losing clearly less win rate as message drop and team size increase; or
  • a higher win rate outright.

Merely matching it under ideal communication would not count as a win.

Ablations. Four ablations isolate the key ingredient, which is linear consensus in a learned lifted space:

  • Replace the linear consensus step with a learned nonlinear message-passing update (a graph network) at identical bandwidth.
  • Average decoded state estimates directly instead of averaging in lifted space.
  • Remove the spectral-radius penalty.
  • Use a hand-built linearization filter where a physics model exists.

Why not just…?

Why not just let agents exchange recurrent hidden states through a learned graph network? This is the most serious threat. Under ideal conditions it may match or beat us, because it is strictly more expressive. Our hypothesis is that it fails differently: its message function has never seen dropped packets at unusual rates or larger graphs, and nothing bounds how errors compound across hops. The first ablation tests exactly this at matched bandwidth. If the graph network is equally robust under drops and team-size changes, the linearity argument fails on practical grounds and we will say so. The certificate then becomes the only remaining contribution.

Why not just add communication to GlobeDiff? We do exactly this in the main comparison. The remaining question is cost and robustness. Diffusion sampling runs many denoising steps per decision, and its fusion of neighbor inputs has no stability structure. We report wall-clock time per decision alongside win rate. On maps where ambiguity is truly multimodal, we expect GlobeDiff to keep its advantage. Identifying those maps is itself a result.

Why not just linearize a physics model and run a distributed extended Kalman filter? On the 4-CSTR chemical reactor benchmark, the parallel Koopman paper reports an RMSE of 1.5204 for Taylor-series linearization against 0.0135 for Koopman-based distributed moving horizon estimation. In SMAC there is no usable analytic model of the opponents anyway. We keep this baseline only in the estimation-only physical testbed.

Why not just use a central server that collects all observations? This is the ceiling on accuracy. We run it as an oracle-style reference. It is ruled out as a deployment design by bandwidth that grows with team size and by a single point of failure. The stress tests quantify that trade-off, and we expect the central server to degrade sharply when its links drop.

Proposed Experiments

Estimation sanity check. We run the distributed estimator on the 4-CSTR reactor and compare with the reported Koopman distributed moving horizon estimation RMSE of 0.0135. Lifting functions are learned here rather than hand-chosen, and message drops are added. No reported numerical result was found on Lacuna for the 96-compartment soil-moisture benchmark, so we use it only to check how estimates scale with 80 of 96 compartments unmeasured.

Main reinforcement learning benchmarks. We use the GlobeDiff partially observed SMAC v1 setting on 6h vs 8z, together with SMACv2. SMACv2 reports that QMIX and MAPPO often fall below 25% win rate on 20-versus-23 maps, and its procedural unit counts allow training on 5-versus-5 and testing on 10-versus-10 and larger. We use the COMRAD Ammo Carrier task to test whether estimator-only centralization avoids the drop the paper reports for MAPPO and HAPPO. No reported numerical result was found on Lacuna for COMRAD Ammo Carrier beyond the statement that independent PPO "significantly outperformed" CTDE methods.

Decisive checks. Three checks decide the claim:

  • Win-rate curves as message-drop probability rises from 0 to 50%, for our estimator, GlobeDiff with communication, and the graph-network ablation.
  • Zero-shot win rate at twice the training team size.
  • The correlation between the measured spectral radius $\rho(M(G))$ on held-out graphs and observed divergence of the estimate.

The proposal succeeds if the lifted-consensus estimator degrades least on the first two checks and the third shows the certificate predicts failures. It fails if the graph-network ablation is equally robust, or if the certificate does not predict divergence.