Many scientific systems can only be observed as unlinked population snapshots. Ocean tracers, plasma particles and dividing cells are measured as clouds of points at a few times, with no record of which point became which. A recent family of models learns the velocity field behind such snapshots and, unlike standard optimal transport, allows that field to rotate. These models are judged almost entirely by how well their predicted snapshots match held-out snapshots, using Wasserstein or Earth Mover's distances, or by pixel-wise error on predicted fields. Those scores have a built-in blind spot. The equation linking density to velocity admits infinitely many velocity fields with identical snapshots, so a snapshot-matching score cannot tell a model that captures a vortex from one that leaves it out.
This is the same kind of failure that image-quality research has studied for a decade. There, mathematically convenient scores disagree with what humans judge, and pixel-wise error rewards blurry averages. We propose to carry that field's diagnostic tools into dynamics modeling. The plan is to build a paired-comparison benchmark whose "judge" is a simulator's ground-truth trajectories rather than a human. We then measure how often current snapshot metrics agree with that judge. Finally, we test whether a metric calibrated on a small budget of tracked trajectories closes the gap, the way LPIPS calibrated deep features on a modest set of human judgments.
Core Claim
Snapshot-based evaluation of non-conservative dynamics models is misaligned with physical truth in two separable ways. The first is gauge blindness. Fields that differ only by a mass-preserving rotation produce identical snapshots, so every marginal metric scores them identically. The second is mean-seeking. On high-dimensional field-valued systems, pointwise error favors blurry conditional-mean forecasts over sharp, physically realistic ones, which is the perception–distortion tradeoff transplanted into physics.
We claim three things can be measured. Current metrics agree with trajectory-level ground truth far less often than assumed. The model rankings reported in this literature can change once rotation-sensitive judgments are used. A small number of tracked particles, used to calibrate a metric, recovers most of the agreement.
The gauge argument is short. Write $\rho_t$ for the population density at time $t$ and $\mathbf{v}$ for a velocity field obeying $\partial_t \rho_t + \nabla\cdot(\rho_t \mathbf{v}) = 0$. Any added field $\mathbf{w}$ with
$$\nabla\cdot(\rho_t\,\mathbf{w}) = 0$$
leaves every $\rho_t$ unchanged, so $\mathbf{v}$ and $\mathbf{v}+\mathbf{w}$ cannot be told apart by any function of the marginals. Leveraging Gauge Freedom for Learning Non-Gradient Population Dynamics of Stochastic Systems uses this freedom as a modeling tool, and the identifiability results in Gradient-flow SDEs have unique transient population dynamics and From geometry to dynamics: Learning overdamped Langevin dynamics from sparse observations with geometric constraints mark where snapshot data stop being informative. We use the same freedom as an evaluation hazard.
How This Relates to Prior Work
The modeling side supplies the systems under test. Curly Flow Matching for Learning Non-gradient Field Dynamics learns a rotating reference path from velocity proxies and transports mass around it. It evaluates with Earth Mover's distance and, on the Taylor–Green vortex, with cosine distance to the true velocity. That second measure is an oracle unavailable on real data. Leveraging Gauge Freedom for Learning Non-Gradient Population Dynamics of Stochastic Systems resolves the gauge with kinetic, curl or divergence penalties. On turbulent tracers it argues for its variants mostly by qualitative trajectory plots, which suggests the authors already sensed that their numbers could not show the difference. A Call to Lagrangian Action: Learning Population Mechanics from Temporal Snapshots learns second-order mechanics. Two-Parameter Flows for Learning Population Dynamics of Physical Systems scales to 128×128 grids. It reports that models fit by pointwise trajectory error fail to reproduce the $\omega^{-3}$ energy spectrum that its own samples match. That is direct evidence for our mean-seeking axis, offered in passing rather than studied. None of these papers asks whether its evaluation metrics can detect the property the method was built to capture. That question is ours.
The image-quality side supplies the method and the theory. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric contributes the two-alternative forced-choice protocol, in which a judge picks the more faithful of two outputs. It also contributes the idea of a judge-agreement ceiling (73.9% for humans on BAPPS) and the recipe of calibrating a feature distance with a small amount of judgment data. We copy the protocol and replace humans with simulator trajectories. The Perception-Distortion Tradeoff and A Theory of the Distortion-Perception Tradeoff in Wasserstein Space predict that error-minimizing estimators blur. The second paper also predicts a quadratic Pareto curve between distortion and realism under its assumptions. We test whether physics emulators sit on that curve. Traversing Distortion-Perception Tradeoff using a Single Score-Based Generative Model supplies variance-scaled sampling, a single knob $\lambda$ that moves one stochastic model from mean-like to sample-like outputs. R-LPIPS: An Adversarially Robust Perceptual Similarity Metric and LipSim: A Provably Robust Perceptual Similarity Metric act as constraints. They show that learned metrics can be gamed by small perturbations, so any calibrated metric we propose must be stress-tested against optimization pressure before it is trusted as a training signal.
We deliberately do not pursue the tempting idea that straight-line flow matching causes image hallucinations such as extra fingers (AGIDefect-4K: A Richly Annotated Dataset for AI-Generated Image Defect Detection, Localization and Explanation). That idea conflates sampling time, the noise-to-data direction, with physical time. Two-Parameter Flows for Learning Population Dynamics of Physical Systems separates those two directions explicitly, and we see no mechanism tying curl in physical time to defects in images.
Research Program
The first component is a benchmark of paired rollouts with trajectory-level ground truth. We use systems where a solver gives full particle paths: the decaying Taylor–Green vortex from LagrangeBench, Kolmogorov and barotropic turbulence on 128×128 grids, two-stream and bump-on-tail Vlasov–Poisson plasma, and HYCOM Gulf of Mexico currents, where tracers can be advected through the model's velocity output. For each system we generate candidate dynamics from several sources: trained models (CURLY-FM, NGIF with each penalty, WLM, TPF, OT-CFM), checkpoints taken across training, and controlled perturbations. The key controls are gauge-equivalent pairs. We build each by adding a field $\mathbf{w}$ satisfying the condition above, which we can construct numerically for a known density. These pairs have provably identical snapshots but different trajectories.
Each benchmark item is a reference simulation and two candidates. The simulator judge picks the more faithful candidate using trajectory-only quantities. These are circulation around closed material loops, separation statistics for pairs of particles, correlations between a particle's positions at two times, and, for field systems, vortex lifetimes. Judges can disagree with each other. We report agreement among the judges as the analogue of the human ceiling, and we do not collapse them into one score.
The second component is a meta-evaluation of existing metrics. For each metric we compute the fraction of pairs on which it agrees with each judge, broken down by whether the pair differs mainly in rotation (gauge pairs), in sharpness (mean-like versus sample-like field forecasts), or in ordinary error. The metrics are the marginal $W_1$, $W_2$ and Earth Mover's distances, pointwise field MSE, energy-spectrum distance, LPIPS on rendered fields, and skewness-augmented feature distances in the spirit of Using Skew to Assess the Quality of GAN-generated Image Features. We then re-rank the published methods using judge agreement. The aim is to find out whether their ordering survives.
The third component is a calibrated metric with a trajectory budget. Following the LPIPS recipe, we compute a small set of candidate features: snapshot statistics, energy spectra, and statistics from $K$ tracked particles or $K$ sparse velocity probes. We fit nonnegative weights to judge preferences on training systems. Evaluation is on held-out systems, so the metric must transfer across physics rather than memorize one flow. The central curve is agreement versus $K$. At $K=0$ the metric is snapshot-only and the gauge argument predicts chance-level agreement on gauge pairs. The question is how quickly agreement climbs as $K$ grows. We then attack the calibrated metric. We optimize a perturbation of the candidate dynamics to improve the metric's score. If the simulator judges do not also improve, the metric is gameable in the sense of R-LPIPS: An Adversarially Robust Perceptual Similarity Metric.
The fourth component tests the distortion–realism curve for physics. On Kolmogorov flow we train one stochastic emulator and sweep variance scaling from $\lambda=0$ to $\lambda=1$, following Traversing Distortion-Perception Tradeoff using a Single Score-Based Generative Model. We plot forecast MSE against a realism score, using spectrum distance and the calibrated metric. We then check whether the frontier follows the quadratic form $D(P)=D^*+[(P^*-P)_+]^2$ from A Theory of the Distortion-Perception Tradeoff in Wasserstein Space, where $D^*$ is the minimum MSE and $P^*$ is the realism score of the MSE-optimal forecast. That paper derives the form under its stated assumptions, and non-Gaussian turbulence is a genuine test of how far those assumptions stretch.
Baselines
The naive reference point is the status quo: rank models by held-out marginal distance ($W_2$ or Earth Mover's), as the CURLY-FM, TPF and WLM papers do. It is computed on exactly the same benchmark pairs. On gauge pairs its expected agreement is 50% by construction, and observing that confirms the benchmark works. On other pairs it is the number to beat. For reference, the WLM paper reports a HYCOM forecasting error of 0.065 against 0.473 for the next best reference-free method (A Call to Lagrangian Action: Learning Population Mechanics from Temporal Snapshots). We will test whether that gap persists under trajectory judges.
The strongest existing snapshot-only competitor is energy-spectrum matching as used by Two-Parameter Flows for Learning Population Dynamics of Physical Systems. It is the only metric in this literature that already separates sharp from blurry forecasts. That paper reports near-perfect matching of the $\omega^{-3}$ spectrum qualitatively, with no numerical spectrum distance found on Lacuna. It is run identically on all field-valued pairs.
The strongest oracle is cosine distance to the true velocity, as reported by Curly Flow Matching for Learning Non-gradient Field Dynamics. Its value is described there only as significantly better than flow matching, and no numeric value was found on Lacuna. It uses the full ground-truth field, so it bounds what a probe-based metric can achieve. We also include LPIPS on rendered vorticity images, to test whether image-perceptual features transfer at all.
The key ablation is the calibrated metric at $K=0$ versus $K>0$, with identical fitting data and regularization. All learned metrics get the same fitting budget on training systems and are evaluated on the same held-out systems. Beating the baselines means two things. On gauge pairs, the metric must clearly exceed 50% agreement with the circulation and pair-separation judges at a modest $K$. On sharpness pairs, it must at least match spectrum distance. The calibrated metric also has to survive the adversarial check.
Why not just…?
Why not just report energy spectra, which already catch blur? On field-valued systems this is a real threat on the sharpness axis, and the benchmark is designed so that spectrum distance may win there; we will report that outcome if it happens. But a spectrum is a single-time statistic of the density or field. It is therefore unchanged by gauge-equivalent rotations, and on particle populations such as tracers, plasma phase space and cells it says nothing about circulation. The gauge-pair split will show whether spectra fail exactly where the argument predicts.
Why not just evaluate with cosine distance to the true velocity, as CURLY-FM does? On real data, which is the point of snapshot methods, the true velocity is unavailable. Velocity proxies such as RNA velocity carry their own biases, a dependence Curly Flow Matching for Learning Non-gradient Field Dynamics itself flags. We keep the oracle as an upper bound. The agreement-versus-$K$ curve then answers the practical question of how much tracking is needed to get close to it.
Why not just build better models, for example by adding curl or divergence penalties? The penalties in Leveraging Gauge Freedom for Learning Non-Gradient Population Dynamics of Stochastic Systems are choices of gauge. Snapshot metrics cannot say which choice is right, which is exactly why that paper fell back on visual trajectory comparisons. Model improvements cannot be credited until an evaluation can see them. The re-ranking experiment tests whether this matters in practice.
Why not just use LPIPS or another image metric on rendered fields? LPIPS is trained on natural-image judgments, is biased toward texture, and is easily pushed around by small perturbations (R-LPIPS: An Adversarially Robust Perceptual Similarity Metric). It is still cheap and might capture vortex sharpness, so it is a measured baseline rather than an assumed failure.
Decisive Checks
The benchmark itself is validated first. Every marginal metric must score gauge-equivalent pairs at chance, while the simulator judges must separate them reliably. If the judges cannot, the trajectory statistics are too noisy and the study stops there.
The main result is the agreement table split by pair type, together with the re-ranking of published methods. A finding that marginal metrics agree with circulation judges near chance on rotation-dominated pairs, and that method orderings change, would show that the field's headline numbers do not measure what its methods claim to capture. The agreement-versus-$K$ curve on held-out systems tells practitioners how many tracked particles to collect. The variance-scaling sweep shows whether physics emulators face the same quadratic distortion–realism frontier as image restoration. If they do not, that marks where the Wasserstein-space theory's assumptions break down.
The calibrated metric must survive the adversarial check before anyone uses it as a training objective. A negative result there is informative too, because it would mean trajectory-level faithfulness cannot be summarized by a cheap learned score.
References
- Leveraging Gauge Freedom for Learning Non-Gradient Population Dynamics of Stochastic Systems
- Gradient-Flow SDEs Have Unique Transient Population Dynamics
- From geometry to dynamics: Learning overdamped Langevin dynamics from sparse observations with geometric constraints
- Curly Flow Matching for Learning Non-gradient Field Dynamics
- A Call to Lagrangian Action: Learning Population Mechanics from Temporal Snapshots
- Two-Parameter Flows for Learning Population Dynamics of Physical Systems
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- The Perception-Distortion Tradeoff
- A Theory of the Distortion-Perception Tradeoff in Wasserstein Space
- Traversing Distortion-Perception Tradeoff using a Single Score-Based Generative Model
- R-LPIPS: An Adversarially Robust Perceptual Similarity Metric
- LipSim: A Provably Robust Perceptual Similarity Metric
- AGIDefect-4K: A Richly Annotated Dataset for AI-Generated Image Defect Detection, Localization and Explanation
- Using Skew to Assess the Quality of GAN-generated Image Features