Satellite maps of sea surface temperature or ocean chlorophyll are full of holes where clouds blocked the view, and neural networks now fill those holes well. They are trained on one period and one grid spacing, and then deployed on later years, when the ocean is warmer or more variable, or on products at a different resolution. This mismatch is rarely tested. Gap-filling has a useful property that most prediction tasks lack. The deployment data supervises itself: you can hide some pixels that were actually observed and ask the network to reconstruct them, with no new ground truth. So adapting a gap filler to a new year or a new sensor is always possible in principle.
The catch is that clouds do not fall at random. Clear-sky pixels differ systematically from cloudy ones; for example, clear skies tend to coincide with particular temperature regimes. If a network is updated freely on re-hidden clear-sky pixels, it can learn that bias. This proposal argues that the question is not whether to adapt but which parameters to let move. Temporal drift should be absorbed by a few normalization parameters computed only from observed pixels. Resolution change should be absorbed by intermediate spatial blocks. Updating everything should reduce error on re-hidden observed pixels while increasing error under real clouds. We would build and test a label-free rule that picks the right parameters for the kind of shift at hand.
Core Claim
Take a pretrained cloud-gap reconstruction network. Adapt it on a short window of deployment data using a masked self-supervised loss. Under these conditions, the location of the updated parameters decides whether adaptation helps. For level and variance drift across years, tuning only the affine parameters of a mask-aware reversible normalization recovers most of the benefit of full fine-tuning. For a change of grid resolution, tuning one intermediate spatial block does the same. Full fine-tuning opens a measurable gap between error on re-hidden observed pixels and error on truly cloud-covered pixels, which is the signature of learned clear-sky bias.
A cheap diagnostic can predict the right parameter group without a validation sweep. The diagnostic is a single pass that measures how much each block's activations or gradients change on the unlabeled deployment window. The novelty lies in three pieces: shift-matched selective adaptation, a gap filler that supervises itself, and clear-sky bias as an explicit failure mode. None of the component methods is new on its own.
Where This Sits in Prior Work
The evidence that where you fine-tune matters comes from outside Earth observation. Surgical fine-tuning for grape bunch segmentation adapted a Mask R-CNN under seasonal, viewpoint and variety shifts. Tuning only the intermediate res3 block reached F1 0.496 under the across-season shift, versus 0.390 for full fine-tuning, while updating 1.22M of 45.3M parameters. That paper found its best blocks by trial and error, on labeled RGB images with no missing data. We keep its central observation and move it to a self-supervised, gappy setting where the risk of over-adapting is sharper.
Layer-wise analysis of fine-tuning and representation landscapes in language models show that adaptation mostly changes upper or post-transition layers. LayerNorm-and-attention tuning for speech translation shows that cross-domain shifts can be absorbed by normalization and attention parameters alone. Together these support our hypothesis that different shift types have different loci in the network. None of them studies spatial fields, resolution, or missingness.
For choosing layers automatically, we reuse the idea from SVD-based layer selection: pick the layer whose representation changes most. We do not use the evolutionary search of P-Transfer, which needs repeated validation sweeps that a label-scarce deployment cannot afford.
The gap fillers themselves supply the backbone and the benchmarks. CRITER 1.0 pairs a Vision Transformer, which forms a coarse field from three days of data, with a UNet that iteratively refines sharp fronts and eddies. It cuts error on missing sea surface temperature pixels by up to 44% relative to DINCAE2 in the Adriatic. It is trained from scratch and evaluated without long-term drift. The partial-physics diffusion model (PPIDM) fills chlorophyll gaps in the multi-year BSOSE simulation. It trains on complete fields and tests on simulated gaps. 3D partial convolutions for soil moisture supply the mask-weighted averaging we use when computing statistics. OxyGenerator confirms that century-scale drift under more than 96% missingness is a real regime, though it uses a custom end-to-end graph model rather than an adapted pretrained one.
The normalization piece builds on two forecasting papers. Reversible instance normalization (RevIN) removes each window's mean and variance before the network and restores them afterward, cutting N-BEATS error on ETTh2 from 6.408 to 0.471 MSE. Non-stationary Transformers warn that normalizing too aggressively flattens attention. Both methods assume dense, gap-free series. Their statistics become biased when computed over zero-filled or cloud-selected pixels, and handling this is part of what we build.
One prior result constrains the design directly. PPIDM showed that forcing incomplete physics hurts: RMSE was 0.538 with full physics versus 0.270 with partial physics. We therefore add no physics loss. Nothing in our claim depends on one, and a physics loss would clash with instance-wise rescaling.
Research Program
Pretrained gap filler. We reimplement a CRITER-style network from its published description, since no code is listed. It has a ViT coarse stage and a UNet refinement stage. We wrap it in a mask-aware reversible normalization. For each variable, the statistics are computed only over valid pixels $M(q)\in\{0,1\}$ in the input window $\Omega$:
$$\mu = \frac{\sum_{q\in\Omega} M(q)\,X(q)}{\sum_{q\in\Omega} M(q)}, \qquad \sigma^2 = \frac{\sum_{q\in\Omega} M(q)\,(X(q)-\mu)^2}{\sum_{q\in\Omega} M(q)}.$$
Here $X(q)$ is the field value at pixel $q$. Inputs are normalized as $\gamma (X-\mu)/\sigma + \beta$ with learnable $\gamma,\beta$, and outputs are mapped back with the same statistics. Because clear-sky pixels are a biased sample, $\mu$ and $\sigma$ are biased as well. The learnable $\gamma,\beta$, and the normalization layers inside the network, are exactly the small parameter group we expect to correct drift.
Self-supervised adaptation. On a short deployment window, such as one season, we draw cloud masks $\tilde M$ from other days. We hide the observed pixels they cover and minimize reconstruction error on those pixels only:
$$\mathcal{L}_{\text{adapt}}(\theta_S) = \sum_{q} M(q)\,(1-\tilde M(q))\,\big(f_{\theta}(X\odot M\odot\tilde M)(q) - X(q)\big)^2.$$
Only the parameter subset $\theta_S$ is updated. The candidate subsets are the normalization parameters, each ViT or UNet block, the output head, and everything.
Shift families. We separate two shift families so that each can be localized. Temporal drift means training on early years and adapting on later ones. Resolution change means pretraining on inputs coarsened by a factor of two and deploying at native grid spacing. A third condition combines the two, which is the realistic case.
Label-free selection rule. For each block, we compute two scores in one pass on the deployment window. The first is the SVD-based representation change between source and deployment activations, following the SVD selection paper. The second is the norm of the gradient of $\mathcal{L}_{\text{adapt}}$, scaled by the block's parameter count. We ask whether the top-scored block matches the best block found by an exhaustive sweep.
Measuring clear-sky bias. In BSOSE the full field is known everywhere. We therefore impose masks that are not random: cloud probability rises with a covariate of the field, such as local chlorophyll or temperature anomaly. This lets us measure error both on re-hidden observed pixels and on truly masked pixels. The difference between the two, as a function of how many parameters were adapted, is the central diagnostic.
Baselines
The naive reference is do nothing: deploy the pretrained gap filler frozen, with its original normalization. Every adaptation method must beat it, and the size of its degradation shows how much drift and resolution change actually cost.
Full fine-tuning on the self-supervised loss $\mathcal{L}_{\text{adapt}}$ is the obvious competitor. It is also the one most likely to win on re-hidden observed pixels. It gets the same deployment window, the same masks, the same number of optimizer steps, and the same learning-rate grid (five values) as every selective variant. It beats our proposal if it matches or improves error on truly masked BSOSE pixels.
Retraining from scratch on the deployment window, as CRITER and PPIDM would be trained, tests whether pretraining matters at all for short windows. It gets matched compute.
Head-only tuning (linear probing) freezes everything except the output layer. Its value is to show whether intermediate spatial features must move under resolution change.
Exhaustive surgical sweep, run per block in the manner of the grape-segmentation paper, tunes each block separately and picks the best on held-out truly masked pixels. It uses ground truth our rule does not have, so it is an oracle upper bound for block selection. The label-free rule succeeds if it lands within 10% of the oracle's error while needing one pass instead of one run per block.
Strongest existing numbers. On the BSOSE infilling task, the strongest reported result is RMSE 0.270 for PPIDM. That result comes from a model trained on complete fields with random gaps and no drift across splits. We will run PPIDM, trained on the early years, as a comparison under our temporal split with non-random masks, and report both its published number and our reproduction. For the real sea surface temperature data, the only reported figure is CRITER's 44% error reduction over DINCAE2 in the Adriatic; no absolute error was found on Lacuna. We will therefore report our reimplementation's error before and after adaptation, alongside DINCAE2 if it can be reproduced. No reported result was found on Lacuna for any gap filler under cross-year drift or cross-resolution deployment.
Ablation of the key ingredient. We compare masked statistics against statistics computed over zero-filled pixels, keeping the tuned parameters fixed. This tests whether mask awareness in the normalization is what makes normalization-only adaptation work.
Why not just…?
Why not just fine-tune everything on the self-supervised loss? On labeled vision data, full fine-tuning already lost to one intermediate block under across-season shift: F1 0.390 versus 0.496 in the grape segmentation study. Our setting adds a specific extra risk. Clear-sky pixels are not representative, so a high-capacity update can fit their statistics. The BSOSE experiment with non-random masks settles this directly. If full fine-tuning shows no growing gap between observed-pixel error and masked-pixel error, the clear-sky bias argument fails, and we will report that.
Why not just recompute normalization statistics on the new data and tune nothing? This is a real threat, especially for pure temporal drift. Mask-aware RevIN with frozen weights might capture almost all of the gain, since RevIN's large gains on ETTh2 came from normalization alone. It is included as its own arm. If it matches normalization-plus-tuning under drift, we will narrow the drift claim to "masked statistics suffice." We still expect it to fail under resolution change, where the shift is in spatial texture rather than in level or variance. The combined-shift condition tests that expectation.
Why not just retrain from scratch on the new period? CRITER-style models are normally trained this way. With years of new data it may well be best. Our regime is a single season. The from-scratch baseline at matched compute on that window measures how much pretraining contributes, and we will also report the window length at which retraining overtakes adaptation.
Why not just resample the input to the training resolution? For resolution change, bicubic resampling of deployment data to the pretrained grid is the simplest fix. CRITER's own motivation is that smoothing erases fronts and eddies, and learnable importance pooling shows that fixed downsampling loses small structures (COCO small-object AP 21.9% versus 24.0%). We run resampling as a baseline and score it on gradient-magnitude error along fronts as well as overall RMSE, because resampling may look fine on RMSE while blurring exactly the features that matter.
Proposed Experiments
Data. The primary testbed is BSOSE, 2008–2012, with 64×64 patches of chlorophyll and velocity. Pretraining uses 2008–2010, adaptation uses one season of 2011, and testing uses 2012. Both random and field-correlated masks are applied at matched coverage. The real-data testbed is the Adriatic, Mediterranean and North Atlantic Level-3 sea surface temperature sets used by CRITER, with the earliest years for pretraining and the latest year held out. Evaluation there follows CRITER's protocol of synthetic masks applied to observed pixels. We acknowledge that true cloud-covered error cannot be measured on real data, which is why BSOSE carries the decisive test.
Metrics. We report RMSE on masked pixels, RMSE on observed pixels, and their gap. We also report error on the gradient magnitude along fronts, using the top decile of the true spatial gradient, and the calibration of any uncertainty map.
Decisive checks. The proposal stands on three predictions. First, under temporal drift in BSOSE, normalization-only adaptation reaches at least 80% of full fine-tuning's reduction in masked-pixel error, and it beats full fine-tuning when masks are correlated with the field. Second, under resolution change, the best single block found by the oracle sweep sits in the refinement UNet's intermediate layers, not in the normalization layers or the head, and the label-free rule picks it in most runs. Third, the gap between observed-pixel and masked-pixel error grows with the number of adapted parameters under non-random masks but not under random masks. If the third prediction fails, the clear-sky bias mechanism is wrong. If the first two show no difference in locus between shift types, the conclusion becomes the simpler "tune one fixed block," and we would report that instead.
References
- Surgical Fine-Tuning for Grape Bunch Segmentation Under Visual Domain Shifts
- Similarity Analysis of Contextual Word Representation Models
- The Representation Landscape of Few-Shot Learning and Fine-Tuning in Large Language Models
- Multilingual Speech Translation from Efficient Finetuning of Pretrained Models
- Enhancing Translation Ability of Large Language Models by Leveraging Task-Related Layers
- Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learning
- CRITER 1.0: a coarse reconstruction with iterative refinement network for sparse spatio-temporal satellite data
- Partial Physics Informed Diffusion Model for Ocean Chlorophyll Concentration Reconstruction
- Generating seamless global daily AMSR2 soil moisture (SGD-SM) long-term products for the years 2013–2019
- OxyGenerator: Reconstructing Global Ocean Deoxygenation Over a Century with Deep Learning
- Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift
- Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting
- LIP: Local Importance-Based Pooling