When we learn the goals of a large population by watching it, for example epidemic avoidance, network patching, or crowd movement, the standard tools assume every demonstrator acts on the same clock. Real populations do not. Some people respond late, some devices patch on slower cycles, and some walkers are simply cautious. These agents can be pursuing exactly the right strategy at a slower pace. Current methods for this setting cannot tell "slow" from "wrong." At a given time step a slow agent occupies states the population as a whole has already left, so the method reads it as a noisy, irrational decision-maker. The recovered reward and policy are then biased toward noise. This note proposes to give each demonstrator a hidden personal speed. Under this model, a demonstrator moves along the shared equilibrium strategy in its own "progress time," while the population distribution that couples everyone stays in physical time. The method then estimates speed and decision quality jointly, so that only real deviations count as suboptimality.
The setting is the mean-field game: a model of very many interacting agents in which each agent responds to the overall distribution of the population (the "mean field") rather than to individual others. Adversarial imitation learns a policy by training a classifier to tell learner behavior from demonstrator behavior and then using that classifier's score as a reward. Prior adversarial inverse methods for mean-field games model imperfect demonstrators only as Boltzmann-rational. A Boltzmann-rational agent picks actions with probability proportional to their exponentiated value, so suboptimality appears as extra randomness over which action is taken. We add a second, structured kind of departure from the canonical schedule: when actions happen. We show how to separate it from the first.
Core Claim
Pace heterogeneity and decision noise leave different fingerprints in population data. A time-rescaled agent produces the same action distribution as the canonical expert at each point of task progress, only spread over more or fewer clock ticks. A noisy agent produces a broader action distribution at each point of progress. A model with a latent per-agent speed can therefore attribute the timing mismatch to pace and the spread mismatch to rationality.
The second half of the claim is specific to mean-field games and is what separates this proposal from simply time-warping trajectories. Some environments have dynamics that depend on the population, such as congestion or contagion. In these environments the mix of speeds changes the physical mean field everyone experiences. Speed cannot then be removed by offline alignment; it has to be modeled inside the population flow. We expect three results. First, pace-blind methods recover rewards whose bias grows with the variance of speeds across the population. Second, offline alignment fails specifically when dynamics depend on the population. Third, the proposed model degrades gracefully in both cases.
Research Program
The mechanism borrows from temporal-scaling work in neuroscience and control. A Recurrent Neural Circuit Mechanism of Temporal-scaling Equivariant Representation shows that a single constant gain input can make a recurrent circuit traverse a fixed path in its state space faster or slower without rewiring. Remington et al. and Kao show that cortex produces different intervals by changing how fast it moves along a fixed trajectory. An Approximate Inference Approach to Temporal Optimization in Optimal Control (AICO-T) supplies the algorithmic template. It introduces a canonical progress variable mapped to physical time by a scaling parameter and fits that parameter by expectation-maximization alongside the trajectory. We reuse the separation between "what path" and "how fast." Our contribution is to place it inside an inverse problem over a population, which none of these works study.
In discrete time we implement speed as a lazy advance. Agent $i$ carries a private progress counter $\tau_i$ and a speed $\alpha_i \in (0,1]$, and at each physical step it advances with probability $\alpha_i$:
$$\tau_i(t+1) = \tau_i(t) + b_{i,t}, \qquad b_{i,t} \sim \text{Bernoulli}(\alpha_i), \qquad a_{i,t} \sim \pi_\beta(a \mid s_{i,t}, \tau_i(t), \mu_t) \text{ if } b_{i,t}=1.$$
Here $s_{i,t}$ is the agent's state, $\mu_t$ is the physical-time population distribution, and $\pi_\beta \propto \exp(\beta Q)$ is a Boltzmann policy with rationality parameter $\beta$. When $b_{i,t}=0$ the agent holds its state, with a lingering transition supplied by the environment. Because holding times are geometric, the variance of the time needed to complete the task grows with its mean. This reproduces the Weber-law signature that the circuit model obtains from multiplicative noise, and it gives us a natural noise model at no cost.
The physical mean field is the mixture over speeds:
$$\mu_t = \int \mu_t^{\alpha}\, dG(\alpha),$$
where $G$ is the population distribution of speeds and $\mu_t^{\alpha}$ is the state distribution of agents with speed $\alpha$. This $\mu_t$ is the quantity that enters the transition kernel and the reward.
Learning alternates two steps. In the E-step, a forward-backward pass over the hold/advance indicators gives each demonstrator a posterior over $\alpha_i$ and over its progress path. In the M-step, an adversarial inverse step updates the reward. For this step we reuse the potential-shaped discriminator of Adversarial Inverse Reinforcement Learning for Mean Field Games (MF-AIRL), which recovers individual rewards under an entropy-regularized Nash equilibrium. We make two changes to it. The discriminator scores only transitions in which the agent advanced, indexed by progress. The sampler's population flow is propagated in physical time under the current estimate of $G$.
We keep the individual-level Nash formulation rather than a population-level social optimum. Individual-Level Inverse Reinforcement Learning for Mean Field Games shows that the social-optimum reduction falls into a "cooperation trap" in competitive games. On Imitation in Mean-field Games points the other way: under population-dependent dynamics, imitation needs a mean-field-control inner loop to avoid error that grows exponentially with the horizon. We treat this as a constraint and not a solved tension. On the population-dependent environment we run both inner loops and report the difference.
On the analysis side, we will state when $\alpha$ and $\beta$ are jointly identifiable. Our conjecture is that they are identifiable whenever the expert's progress-indexed action distribution is not uniform and each demonstrator contributes enough advance steps. We will also characterize how the error of a pace-blind estimator grows with the variance of $G$.
Baselines
The naive reference point is behavior cloning on states augmented with normalized time $t/T$. It is cheap, and it shows how much the problem is really about compounding error. On Imitation in Mean-field Games proves that this kind of cloning compounds error exponentially when dynamics depend on the population, so we expect it to fail there. It may be competitive on the reward-coupled tasks, and we will report it either way.
The strongest existing competitor is MF-AIRL Adversarial Inverse Reinforcement Learning for Mean Field Games. It is the only method in the file that is adversarial, mean-field, and models suboptimal demonstrators. It reports outperforming the earlier mean-field inverse methods PLIRL and MFIRL on INVEST, MALWARE, VIRUS, RPS and LR in policy KL deviation and expected-return gap. We found no reported numerical value for MF-AIRL on Lacuna, so we will rerun it ourselves. It will use the same demonstration sets, the same discriminator and policy architectures, the same number of environment rollouts, and an equal hyperparameter search budget, including a search over its entropy temperature. On the Attractor environment we also run MFC-ADV from On Imitation in Mean-field Games, the strongest method designed for population-dependent dynamics. We found no reported numerical result for its Nash Imitation Gap on Lacuna.
Beating these baselines means a lower reward-recovery error and a lower Nash Imitation Gap of the learned policy than the best-tuned MF-AIRL. The gain must be statistically separated across seeds and must grow with the variance of speeds. The method must also match MF-AIRL when all speeds are equal.
Three further baselines isolate the key ingredient. The first is MF-AIRL run on demonstrations pre-aligned by dynamic time warping. The second is MF-AIRL with an ILEED-style per-demonstrator expertise weight, following Imitation Learning by Estimating Expertise of Demonstrators. The third is an ablation of our own model that keeps latent speeds but propagates the mean field in progress time rather than physical time; this tests whether the population coupling matters. An oracle that is given true $\alpha_i$ bounds what speed inference can deliver.
Why not just…?
Why not just add the time index, or a measured speed, to the policy input? If speed were directly observed, conditioning on it would be a strong and simple solution, and our oracle baseline is exactly that. The hard case is the one where speed is latent and confounded with rationality, as with crowdsourced or latency-affected demonstrators. Conditioning on wall-clock time cannot remove the confound because the same clock time maps to different progress for different agents. We will measure the gap between latent inference and the oracle. If that gap is negligible and speed can be cheaply estimated from trajectory length, we will say the contribution reduces to that heuristic.
Why not just align trajectories with dynamic time warping and run MF-AIRL? Warping aligns each trajectory geometrically and ignores the population. When transitions depend on the mean field, a slow agent truly experiences a different population at a given progress point. Warping its trajectory onto the canonical schedule then creates transitions the environment would not produce. The progress-time mean-field ablation directly tests this. We predict that warping roughly matches our method on reward-coupled tasks such as RPS and fails on the Attractor environment and Night Club. If warping holds up on those environments as well, the mean-field half of our claim is wrong.
Why not just weight demonstrators by estimated expertise, as ILEED does? ILEED models flaws as action noise scaled by a latent expertise. A slow but correct agent would be down-weighted, which throws away valid evidence about the reward and biases the population estimate toward fast agents. Its own description leaves time-lagged execution unaddressed. We expect expertise weighting to look reasonable on policy return but to recover rewards worse than our method when speed and competence are uncorrelated. The experiment in which we vary that correlation settles this.
Proposed Experiments
We generate ground-truth demonstrations on the five synthetic tasks used by MF-AIRL and MFIRL, on the Attractor environment from On Imitation in Mean-field Games, and on Beach Bar and Night Club from Population-Aware Imitation Learning in Mean-field Games with Common Noise. We compute entropy-regularized equilibria under a known reward. Then, independently, we vary two factors: the spread of $G$ (all speeds equal, up to a fourfold range) and the rationality $\beta$. We also vary whether speed and $\beta$ are correlated across demonstrators. The metrics follow the source papers: KL policy deviation, expected-return gap, mean-field deviation, and exploitability or Nash Imitation Gap. We add correlation between recovered and true reward, and calibration of the inferred $\alpha_i$.
The decisive checks are four. Pace-blind MF-AIRL should show a fitted rationality that falls as speed spread grows even though true $\beta$ is fixed; this is the bias we claim exists, and if it is absent the motivation fails. Our method's recovered $\beta$ should stay flat. Dynamic time warping should break specifically on population-dependent dynamics. When all speeds are equal, our method should not lose to MF-AIRL.
For real data, we use the Twitter topic dataset from Learning Deep Mean Field Games for Modeling Large Population Behavior. There, user-level activity rates plausibly vary, and hourly binning is exactly the setting where Neural Network Poisson Models for Behavioural and Neural Spike Train Data warns of boundary averaging artifacts. The reference point is that paper's deep mean-field model, which reported 58% lower end-of-day distribution error and 40% lower average hourly error than vector autoregression. We will report whether latent speeds reduce held-out distribution error further. We treat this as supporting evidence, since no ground-truth reward exists. A secondary single-agent check on Robomimic Square, whose human operators are binned Worse/Okay/Better, asks how much of the "Worse" label is explained by speed alone versus by deviations in which actions were chosen.
References
- A Recurrent Neural Circuit Mechanism of Temporal-scaling Equivariant Representation
- Flexible Sensorimotor Computations through Rapid Reconfiguration of Cortical Dynamics
- Measurement, manipulation and modeling of brain-wide neural population dynamics
- An Approximate Inference Approach to Temporal Optimization in Optimal Control
- Adversarial Inverse Reinforcement Learning for Mean Field Games
- Individual-Level Inverse Reinforcement Learning for Mean Field Games
- On Imitation in Mean-field Games
- Imitation Learning by Estimating Expertise of Demonstrators
- Population-Aware Imitation Learning in Mean-field Games with Common Noise
- Learning Deep Mean Field Games for Modeling Large Population Behavior
- Neural Network Poisson Models for Behavioural and Neural Spike Train Data