One Latent Space, Many Sizes: Choosing the Assimilation Dimension On the Fly

Fast data assimilation means correcting a running simulation with sparse sensor readings. A popular recipe compresses the physical state with an autoencoder, a network trained to squeeze a field into a short vector of numbers, its "latent" code, and to reconstruct the field from that code. A small learned model then advances the code in time, and a Kalman-style ensemble filter corrects the code whenever observations arrive. The single most consequential design choice in this recipe, the size of the latent code, is currently fixed by hand once and for all. LD-EnSF uses 10 latent dimensions for 2D Kolmogorov turbulence regardless of whether 0.1% or 5% of the domain is observed. Choosing it properly would mean training a separate autoencoder and dynamics model for every candidate size, each on expensive simulator trajectories, and then running the filter on each. That is an architecture search paid for in both GPU time and simulation data.

We propose to remove that search. We train a single latent model whose coordinates are ordered by importance, so that the first $k$ coordinates are always a usable $k$-dimensional model. At each assimilation cycle we then pick $k$ from the data. The motivation is a trade-off that the latent assimilation literature has not exploited. An ensemble filter with a fixed number of members estimates covariances worse as the state it tracks gets bigger. A too-small code, on the other hand, cannot represent the true state. The best size should therefore move with ensemble size and observation density, and an ordered latent code lets us follow it at no extra training cost. Each ensemble member is a surrogate rollout, so this also reduces the number of rollouts the filter needs. That is where the sample-complexity cost of assimilation sits.

Core Claim

Our claim has three parts. First, importance ordering can be imposed on the latent bottleneck of a latent-dynamics model with little loss relative to separately trained fixed-size models. Second, the analysis error of a latent ensemble filter, meaning its error after an observation update, is minimized at a latent size $k^*$ that grows with ensemble size $N$ and with sensor coverage. Third, choosing $k$ per cycle from held-out sensor residuals beats any single fixed size, including the hand-chosen sizes in prior work, at small ensemble sizes and extreme sparsity.

The reasoning behind the second part can be written as a rough error decomposition. Here $\varepsilon_{\text{trunc}}(k)$ is the decreasing error from representing the state with only $k$ coordinates, and the second term is the sampling error of estimating a $k$-dimensional covariance from $N$ members, which scales like $k/N$ for sample covariances:

$$ \mathbb{E}\,\|x - \hat{x}^a_k\|^2 \;\approx\; \varepsilon_{\text{trunc}}(k) \;+\; c\,\frac{k}{N}. $$

In this expression, $x$ is the true state, $\hat{x}^a_k$ is the decoded analysis using $k$ latent coordinates, and $c$ depends on observation noise and coverage. The optimum $k^*$ is the point where the marginal drop in truncation error equals $c/N$. We do not need the constant $c$ to be known, because $k$ is chosen empirically.

Research Program

Ordered latent dynamics model. We take the improved latent-dynamics network of LD-EnSF, which is a ResNet with Fourier-feature encoding, two-stage training, and an LSTM encoder that aggregates sparse observation histories. We widen its bottleneck to a maximum size $K_{\max}$, for example 64 instead of 10, and borrow the soft importance ordering from Adaptive Width Neural Networks. That paper applies a monotonically decreasing scaling over hidden units under a variational objective, which is what lets trailing units be pruned with little accuracy loss. AWN treated only the hidden widths of multilayer perceptrons and never an assimilation state, and it lists extension beyond MLPs as open. We apply its ordering to the latent coordinates. During training we also randomly truncate the code to a sampled $k$ before decoding and before the multi-step rollout loss. The rollout loss is the chained loss that Latent Space Data Assimilation by using Deep Learning showed prevents drift. Training on truncated codes forces every prefix to be a self-consistent dynamical model, not merely a good reconstruction.

Filter with per-cycle size selection. The filter keeps an $N$-member ensemble of length-$K_{\max}$ codes. Our main filter is a latent ensemble transform Kalman filter in which the decoder serves as the observation operator, following ETKF-Q-L. We also use the latent score filter of LD-EnSF to check that the result does not depend on the filter type. At each cycle we compute candidate analyses for a grid of $k$ values. For a given $k$, the trailing coordinates are held at the ensemble mean and the update acts only on the first $k$. All of this is done in latent space, so the cost is negligible compared with decoding.

We choose $k$ by cross-validation over sensors. A random 20% of the cycle's sensors is withheld from the update, and we select

$$ \hat{k}_t = \arg\min_k \sum_{j \in \mathcal{V}_t} \big(y_j - h_j(\bar{z}^a_{k,t})\big)^2. $$

In this rule, $\mathcal{V}_t$ is the set of withheld sensors at time $t$, $y_j$ is the reading of sensor $j$, $h_j$ is the decoder followed by the observation at that sensor's location, and $\bar{z}^a_{k,t}$ is the analysis mean computed with $k$ coordinates. After selection, the update is rerun with all sensors at $\hat{k}_t$.

Analysis. We will measure $k^*$ as a function of $N$ and coverage, and test whether it follows the predicted trend: growing with $N$ and with the number of sensors. We will also characterize the failure regime, namely whether the selection rule chooses too-small codes when held-out sensors are so few that the residual itself is noisy. That regime is exactly the 0.1% coverage setting that LD-EnSF targets.

Baselines

The do-nothing reference is a latent filter with a fixed code size equal to the hand-chosen value in prior work: 10 for Kolmogorov flow, as in LD-EnSF, and 40 for augmented Lorenz 96, as in ETKF-Q-L. It is trained on the same trajectories with the same epochs. Beating it shows that the hand-chosen size is not the right size at every sparsity level and ensemble size.

The strongest existing method on the benchmark that has published numbers is ETKF-Q-L from Latent Space Data Assimilation by using Deep Learning. It reports a mean RMSE of 0.168 on augmented Lorenz 96, against 0.194 for the full-space filter and about 0.383 for a PCA reduction. We will reproduce it on that benchmark with the paper's ensemble size and observation setup, and then rerun it at smaller ensemble sizes. Matching 0.168 at the paper's settings is a sanity check. Our method beats ETKF-Q-L if, at equal ensemble size, it achieves lower RMSE at the published operating point, or matches that RMSE with half the members. On Kolmogorov flow, the shallow-water tsunami case and PlanetSWE, LD-EnSF is the strongest competitor. We found no reported result on Lacuna for LD-EnSF's assimilation error on these benchmarks, only speedups and sparsity levels, so we will rerun it ourselves with its 10-dimensional latent code under our protocol.

The expensive oracle is a grid of separately trained fixed-size models, one per $k$, each trained with the same per-model budget. For each condition we select the best one in hindsight. This is the architecture search we claim to eliminate, so we report its total GPU hours and simulator trajectories alongside its accuracy.

The ablation that isolates ordering keeps the same ordered model but fixes $k$ to the single size that is best offline, averaged over all conditions. Any gain of the adaptive method over this ablation comes from per-cycle selection. Any gain of the ablation over the do-nothing reference comes from ordering plus a better fixed choice.

All methods use identical training trajectories, observation noise, sensor locations and number of filter cycles. Learned components receive the same hyperparameter tuning budget, and the same inflation and localization tuning is given to every filter.

Why not just…?

Why not just use POD/PCA, which is already ordered for free? This is the closest cheap competitor, since a proper orthogonal decomposition basis supports exactly the same per-cycle truncation. The evidence from ETKF-Q-L, where PCA gave an RMSE of about 0.383 against 0.168 for the nonlinear latent code, comes from a fixed size. It does not rule out that adaptive truncation rescues PCA. We therefore run ordered POD with our identical selection rule as a baseline. If it matches the ordered neural code on Kolmogorov flow, then the contribution is the selection rule alone, and we will report it that way.

Why not just keep a large fixed latent code and fix small-ensemble problems with inflation and localization? This is the standard remedy in ensemble assimilation and a real threat, because tuned inflation partially absorbs sampling error. We give the fixed-$K_{\max}$ filter a full sweep of multiplicative inflation and latent localization radii, tuned on validation trajectories, which is a budget our method does not receive. The claim fails if the tuned large filter matches adaptive selection at $N \le 16$.

Why not just train a model per size and pick the best, as an ordinary search? At matched accuracy it may be marginally better per size, so this baseline asks what the ordering costs. We report per-$k$ reconstruction and filter error of the single ordered model against each separately trained model. Our success criterion is staying within 10% of the separate models at every $k$, while using one training run and one set of trajectories. An offline search also cannot change $k$ as coverage changes during a deployment, which our moving-coverage experiment tests directly.

Why not use a zero-cost proxy to pick the size before training? Proxies such as EZNAS reach a Kendall tau of 0.65 on CIFAR-10 classification. They were evolved for convolution blocks and classification loss, and they lose ranking accuracy among the top candidates, which is exactly where latent-size choices lie. More fundamentally, the best size here depends on $N$ and on sensor coverage at deployment, and no score computed at initialization sees either quantity.

Proposed Experiments

Augmented Lorenz 96. The first benchmark is augmented Lorenz 96, a 40-variable model mapped nonlinearly into 400 dimensions, following ETKF-Q-L. Here the true latent size is known to be 40, which allows a clean check that $\hat{k}$ approaches 40 at large $N$ and falls below it at small $N$. We sweep $N \in \{8, 16, 32, 64, 128\}$ and observation fractions from 5% to 50%.

Fluid and ocean benchmarks. The second benchmark is 2D Kolmogorov flow at 0.1%, 1% and 5% coverage, using the LD-EnSF protocol. The third is a shallow-water case drawn from The Well via PolymathicAI/the_well. On this case we also run a moving-coverage scenario in which sensors drop out mid-trajectory. A fixed size cannot react to this change, and per-cycle selection should.

Metrics. We measure analysis RMSE over the field, continuous ranked probability score for ensemble calibration, and the time series of $\hat{k}_t$. We also count costs: simulator trajectories, training GPU hours, and surrogate rollouts per cycle.

Decisive checks. Three outcomes would settle the proposal. First, a monotone dependence of the empirically best $k$ on both $N$ and coverage would confirm the trade-off, while a flat $k^*$ would refute it and make the method pointless. Second, at $N \le 16$ the adaptive filter should beat both the hand-chosen fixed size and the inflation-tuned large filter on RMSE and calibration. Third, the single ordered model should stay within 10% of the per-size grid while costing roughly one $k$-th of its training compute and data.

If the first check fails, we learn that latent filter error is dominated by model error rather than sampling error at these scales. That would be useful to know before anyone invests in searching latent architectures for assimilation.