When a team of controllers trained in simulation is deployed on a real system, such as a grid of traffic signals or several robots sharing a workspace, the simulator's physics are wrong in ways that hurt every agent at once. One established fix is grounding: learn a small correction, called an action transformer, that adjusts each simulated action so the simulator's transitions match what the real system does. The best multi-agent grounding method needs a log of every agent's real actions. The best action-free grounding method handles only one agent. Real deployments often record only states, for example vehicle positions from cameras or queue counts from loop detectors, with no record of what each actuator did.
This note proposes a way to ground a multi-agent simulator from state-only real logs. The core move is that, while learning agent $i$'s correction, every neighbor is pinned to its logged real next state. The problem then splits into $n$ separate single-agent problems whose targets do not move. This removes the "cascading invalidation" that makes simultaneous multi-agent grounding unstable. It also removes the misplaced blame that arises when a single judge scores an agent for its neighbors' behavior. Pinning neighbors creates a train-versus-deployment mismatch, which is the classic covariate shift of imitation learning. A short closed-loop phase in which neighbors move freely corrects it. That second phase is where covariate shift enters this proposal, and the experiments measure it directly.
Core Claim
Two prior works define the gap.
GARAT showed that single-agent grounding is an imitation-from-observation problem. An adversarial discriminator compares grounded-simulator transitions $(s,s')$ with target transitions, and the action transformer is trained by reinforcement learning to fool it. With one target trajectory it achieved lower transition error than GAT-style grounding achieved with 100.
JL-GAT extended grounding to many interacting traffic-signal agents. It conditions forward and inverse models on neighbors' states and actions within a radius. It names the cascading invalidation effect: when all agents update their grounding at once, each agent's learned correction becomes stale. JL-GAT damps this with checkerboard and probabilistic update schedules. It requires target-environment action records, and it composes separately trained forward and inverse models, which RGAT found to compound errors when policies are deep networks.
The obvious combination would run GARAT per agent with JL-GAT's checkerboard schedule. That treats the symptom. Our claim is different. When real neighbor states are available in the log, the source of non-stationarity can be removed rather than scheduled around. If the neighbors' next states are clamped to logged values, agent $i$'s grounding target is a fixed data distribution, whatever the other correctors are doing. Grounding then becomes stationary, embarrassingly parallel, and blame-free.
Concretely, let $s_i$ be agent $i$'s local state, $s_{N(i)}$ the states of its neighbors, and $g_{\phi_i}(s_i, s_{N(i)}, a_i)$ the learned correction to agent $i$'s action $a_i$. The action $a_i$ is sampled from the known deployed policy $\pi_i$; only the real system's realized actions are unrecorded. In the grounded simulator, agent $i$ steps with the corrected action while neighbors are set to their logged next states $\tilde{s}'_{N(i)}$. We train $\phi_i$ to make the conditional transition distributions match:
$$\min_{\phi_i}\; D\Big(\rho^{\text{sim}}_{\phi_i}\big(s_i, s_{N(i)}, \tilde{s}'_{N(i)}, s'_i\big)\;\big\|\;\rho^{\text{real}}\big(s_i, s_{N(i)}, s'_{N(i)}, s'_i\big)\Big),$$
Here $\rho$ is the occupancy measure over local transitions, the long-run frequency with which each transition is visited, and $D$ is estimated by a discriminator as in GARAT.
Conditioning on the neighbors' realized next states is what separates "my dynamics are off" from "my neighbor did something." This is exactly the confusion DecompGAIL calls irrelevant interaction misguidance: a single discriminator over the whole scene penalizes an agent for collisions between other agents. DecompGAIL fixes it architecturally by splitting the discriminator into ego-map and ego-neighbor parts, with action labels available. We fix it by conditioning on data, with no actions.
Research Program
The method has two phases.
Phase one (clamped grounding). Run clamped grounding in parallel for all agents on state-only logs. The logs are collected by deploying the current source-trained stochastic policies on the target system and recording states only. Because each $\pi_i$ is known, any transition mismatch is attributable to dynamics rather than behavior. This identifiability argument is the reason the setting is well posed, and it fails if the real logs come from an unknown controller.
Phase two (closed-loop correction). Release the clamps and roll the full grounded simulator forward step by step. Fine-tune all correctors jointly against a whole-scene transition discriminator for a small number of iterations. This follows the stepwise forward-simulation logic that ForSim used to remove compounding drift in multi-vehicle policies.
Phase two exists because clamping is a form of teacher forcing. In free rollout, neighbors' next states come from the grounded simulator, not the log. Small per-agent errors can then compound jointly, as Rajaraman et al. formalize with an $O(H^2)$ horizon dependence for open-loop imitation. We expect phase two to need far fewer iterations than unclamped joint grounding, because it starts from correctors that are already locally correct.
After grounding, policies are retrained in the grounded simulator and redeployed. The loop can repeat.
Analysis. The main analysis is an invalidation probe. Freeze agent $i$'s corrector, update all neighbors' correctors, and measure the change in agent $i$'s held-out transition error. JL-GAT's argument predicts this change is large for unclamped simultaneous grounding. Our method predicts it is near zero during phase one by construction. The open question is how much invalidation phase two reintroduces.
Constraint from game theory. Matching Multiple Experts proves that state-only occupancy matching can leave agents highly exploitable when data lack full support, unless the game has smooth best responses. This constrains our scope. We target cooperative or traffic-like settings where grounding error, not strategic exploitation, is the dominant failure. We report support coverage of the logs as a diagnostic rather than claim guarantees for competitive games.
Baselines
Direct transfer (naive reference). Deploy policies trained in the ungrounded source simulator. On the CityFlow-to-SUMO 4x4 Manhattan grid under rain, JL-GAT reports an average travel time of 485.63 s for direct transfer. Any grounding method must clearly improve on this.
JL-GAT with real target actions (strongest existing method). JL-GAT is the only published multi-agent grounding method in the file, and it is designed for exactly this benchmark. It reports 467.11 s on the 4x4 rainy grid and 456.14 s on the 1x3 snowy grid. We give it the action logs it requires, so it acts as an upper reference that has privileged information. We use the same target-data budget, measured in logged steps; the same policy architecture and retraining budget; and an equal hyperparameter search for every method. Our method succeeds if, without any actions, it lands within the run-to-run spread of 467.11 s on the 4x4 grid. Beating it outright would be notable, and plausible only because we avoid composing separate forward and inverse models.
JL-GAT with inferred actions. We train an inverse dynamics model on source-simulator data to label the state-only target logs, then run JL-GAT on those labels. This is the natural action-free version of the strongest method. The criterion is that our method beats it in travel time and degrades less as target logs shrink.
Per-agent GARAT. We run GARAT per agent with simultaneous updates, and also with JL-GAT's checkerboard schedule. This is the obvious combination. It isolates whether clamping beats scheduling as a cure for invalidation.
Centralized GARAT. A single action transformer over the joint state, which JL-GAT reports scaling poorly.
Ablations. Two ablations isolate the key ingredients. The first removes clamping, conditioning on neighbor states $s_{N(i)}$ but letting neighbors evolve in simulation. The second removes phase two, using clamped grounding only.
Domain randomization. Policies trained over randomized weather-like dynamics parameters. This is treated as a separate baseline below.
Why not just…?
Why not just label the logs with an inverse dynamics model and run JL-GAT? This is the most credible threat. IOSTOM reports that bypassing inverse dynamics outperforms inverse-dynamics-based imitation on 23 of 24 D4RL tasks, especially with few expert trajectories and with mismatched dynamics. RGAT shows that composing separate learned dynamics models breaks with deep policies. In our setting the inverse model is trained on source physics and applied to target transitions. Its labels are therefore biased in precisely the direction grounding is meant to correct. Still, in signal control the action space is small and discrete, so inverse labeling may be accurate. The inferred-action JL-GAT baseline, swept over target-log size, settles this. If it matches us at every size, the clamping contribution reduces to a simplification, and we would report that.
Why not just run GARAT per agent with JL-GAT's checkerboard schedule? Scheduling only slows invalidation. Each agent still trains against neighbors whose correctors change between its updates. It also leaves misattribution untouched: an unconditioned per-agent discriminator still sees neighbor-induced transitions as the agent's own error, the failure DecompGAIL documents. The invalidation probe and the no-clamping ablation show directly whether the gain comes from clamping or merely from per-agent decomposition.
Why not just randomize the simulator's dynamics? The sim-to-real literature summarized in the file reports that broad randomization yields conservative policies. RoboTwin shows tightly coordinated tasks still below 15% success despite heavy synthetic variation. Randomization also uses none of the real logs. We include it with a randomization range tuned on the same target data budget. If it matches grounding on the traffic grids, that suggests the CityFlow-to-SUMO weather gap is too simple, and the robot benchmark below becomes the deciding test.
Proposed Experiments
Primary benchmark. The primary benchmark is CityFlow-to-SUMO from JL-GAT, on the 1x3 and 4x4 grids under rain and snow. The SUMO logs are stripped to state-only queue and vehicle observations. The metric is average travel time, with reported anchors of 485.63 s for direct transfer and 467.11 s for JL-GAT on the 4x4 rainy grid. We sweep the number of logged target episodes downward to test GARAT's few-trajectory advantage in the multi-agent case. Alongside travel time we report the invalidation probe and per-agent transition error.
Continuous control. The second test uses a multi-agent version of the modified-MuJoCo dynamics-mismatch benchmark used by GARAT: two or more coupled bodies sharing contacts, with mass and friction shifts in the target. There is no reported result on Lacuna for a multi-agent variant, so all methods are run fresh, and target-trained policies serve as the ceiling.
Scaling to uncontrolled neighbors. Logged Waymo Open Motion Dataset tracks serve as a state-only "real" target, with background vehicles clamped to their logs, as an exploratory test of grounding with human neighbors whose actions are never available. Here the realism meta-metric of the Sim Agents Challenge is the evaluator; log playback scores 0.722. DecompGAIL's 0.7864 was obtained with actions on a different task, policy learning rather than grounding, so it is context only.
Decisive checks. Three outcomes would decide the question. First, clamped grounding should show near-zero invalidation during phase one while unclamped variants do not. Second, removing phase two should hurt most on the dense 4x4 grid, where compounding joint drift is largest. Third, the method should reach JL-GAT's action-labeled travel time without actions, and stay ahead of inverse-model labeling at small log sizes. If the third check fails while the first two hold, the contribution narrows to a cleaner diagnosis of invalidation. If the first check fails, the central claim is wrong.
References
- An Imitation from Observation Approach to Sim-to-Real Transfer
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- Joint-Local Grounded Action Transformation for Sim-to-Real Transfer in Multi-Agent Traffic Control
- DecompGAIL: Learning Realistic Traffic Behaviors with Decomposed Multi-Agent Generative Adversarial Imitation Learning
- ForSim: Stepwise Forward Simulation for Traffic Policy Fine-Tuning
- Toward the Fundamental Limits of Imitation Learning
- Matching Multiple Experts: On the Exploitability of Multi-Agent Imitation Learning
- IOSTOM: Offline Imitation Learning from Observations via State Transition Occupancy Matching
- RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins
- The Waymo Open Sim Agents Challenge