Most of the world's rivers have no gauge. Rainfall and temperature, however, are known almost everywhere from gridded weather products. Hydrologists have tried two routes to good forecasts when river-flow labels are scarce. The first splits the flow record into slow and fast frequency components and forecasts each separately. The second pretrains a network on unlabeled weather data and then fine-tunes it on the few gauges that exist. Each route has a blind spot. The frequency-splitting methods need a long local flow record to decompose, which ungauged basins do not have. They are also usually run in a way that lets future data leak into training. The pretrained models treat weather as raw tokens and score reconstruction with a plain squared error, and that error is dominated by individual storms. A catchment's response to rain depends on how much water it already stores, and storage is set by the slow weeks-to-months part of the weather. The standard loss gives that slow part almost no weight.
This note proposes moving the frequency split out of the preprocessing pipeline and into the self-supervised loss. The network is pretrained on weather alone. It reconstructs masked spans of forcing, and its error is measured separately on a small set of causal, one-sided smoothed versions of each variable, which act like leaky water buckets with time constants from days to seasons. Each band's error is normalized so the slow, storage-like bands count as much as the storm band. The model is then fine-tuned on a handful of gauges. The claim is that this specific weighting, not pretraining in general, produces most of the gain when labels are scarce in both space and time and when the test climate differs from the training climate.
Core Claim
The obvious combination of these two literatures is "self-supervised pretraining of decomposed sub-signals," and nothing in the file does exactly that. The literal version is a poor idea, though, and seeing why shapes the proposal.
Decomposition-hybrid models report very large gains. EEMD-LSTM reaches NSE 0.98 against 0.91 for a plain LSTM on Three Gorges inflow (Daily Runoff Forecasting Using Ensemble Empirical Mode Decomposition and Long Short-Term Memory). A CEEMDAN-VMD pipeline cuts dissolved-oxygen RMSE from 0.269 to 0.036 (Application of an improved LSTM model based on FECA and CEEMDAN VMD decomposition in water quality prediction). (NSE, the Nash-Sutcliffe efficiency, is 1 for a perfect forecast and 0 for always predicting the mean flow.) These pipelines decompose the target series itself. When the decomposition is done once over the whole record, future values leak into past components; when it is redone online at every step, it becomes expensive and its gains are untested. Either way, the method cannot run in a basin with no flow record (A Data Scientist's Guide to Streamflow Prediction). This literature motivates the idea that frequency structure matters, and it constrains us: the decomposition must be causal and must not touch the target.
Our version applies a fixed causal filter bank to the forcings only. Each filter is an exponential moving average (EMA) of a forcing variable with time constant $\tau_b$, and each band is the difference of two adjacent EMAs. An EMA of precipitation is exactly the storage of a linear reservoir with residence time $\tau_b$. Reconstructing these bands therefore asks the encoder to recover antecedent-wetness proxies, the quantity that Kan et al. identify as controlling whether the same rain produces a small rise or a flash flood. The pretraining loss is
$$ \mathcal{L} = \sum_{b=1}^{B} \frac{1}{\sigma_b^{2}} \sum_{t \in M} \left\| F_b(\hat{x})_t - F_b(x)_t \right\|^{2}. $$
Here $x$ is the multivariate forcing series of a basin, $\hat{x}$ is the network's reconstruction, $M$ is the set of masked time steps, $F_b$ is the causal band-$b$ filter, and $\sigma_b^{2}$ is the variance of band $b$ over the pretraining corpus. Dividing by $\sigma_b^2$ gives each band equal expected weight. Without it the storm band dominates, because daily precipitation variance sits overwhelmingly at high frequency.
Masked spans are long (30 to 120 days) and cover whole variable groups. A slow band therefore cannot be filled in by interpolating across the gap. It has to be inferred from the static attributes and the other variables, which avoids the "task too easy, model learns local autocorrelation" failure reported for wearable foundation models (Scaling Wearable Foundation Models).
Relation to Prior Work
The closest existing work is StefaLand (Structured Physical Attributes Enable Efficient Foundation Models for Land-Surface Prediction). It is a 12M-parameter masked autoencoder, meaning a network that learns by filling in hidden parts of its input. It hides whole groups of variables across 8,634 catchments and reports a 20% RMSE reduction over supervised LSTMs on ungauged basins. We reuse its group masking and its residual decoder design. We change two things: the loss is band-separated and storage-weighted instead of a raw-token squared error, and the evaluation crosses label scarcity with climate shift, which StefaLand does not.
AutoCon (Self-Supervised Contrastive Learning for Long-term Forecasting; Self-Supervised Contrastive Forecasting) pairs windows that share high linear autocorrelation, for example windows 365 days apart, and trains a long-term branch to treat them as similar. Its authors concede it degrades when cycles drift. It was also never tested on transfer to unlabeled systems. Our objective uses no calendar pairing, so drifting seasons do not create false positives.
The ECG study of Mehari & Strodthoff supplies a methodological prior. There, a predictive objective (CPC, contrastive predictive coding) beat instance discrimination, and the gains were largest on rare classes. That leads us to expect the largest gains on low flows, where Gauch et al. found that more labeled data barely helps.
The supervised global LSTM of Nearing et al. and CauSTream (CauSTream) are both fully supervised. CauSTream learns separate causal graphs for runoff generation and river routing. We do not model routing, and we treat both systems as context, not competitors. h-Diffusion (Diffusion-Based Probabilistic Modeling for Hourly Streamflow Prediction and Assimilation) needs an external daily model as input and roughly 12 hours of inference for 5 years of assimilation. Our model runs in one forward pass.
Research Program
The build has three parts.
The first is a filter bank of roughly five bands, with time constants from about 2 to 180 days, applied to precipitation, temperature, radiation and vapor pressure. It is computed online at negligible cost.
The second is a Transformer encoder of StefaLand's size (about 12M parameters) that takes static catchment attributes and forcing tokens. It is pretrained with the loss above on forcing from every basin in CAMELS-US, CAMELS-GB and CAMELS-Chile, with no streamflow. This includes the basins later held out as "ungauged," which is legitimate because their weather is observable in deployment.
The third is fine-tuning with an LSTM decoder on streamflow from a chosen subset of gauges.
The analysis asks three questions. Does band weighting beat uniform weighting at matched pretraining? Is the gain concentrated in the scarce-label, shifted-climate corner? Do the frozen encoder's band activations track storage? The last question is checked against a calibrated bucket model's soil moisture, in the spirit of CauSTream's comparison with the VIC process model, and is a sanity check only.
Baselines
The naive reference points are the mean-flow forecast, which scores NSE 0 by definition, and one-day persistence. Every model must clear persistence at one-day lead time before any other comparison means anything.
The main supervised competitor is the Entity-Aware LSTM (EA-LSTM), an LSTM whose gates are conditioned on static catchment attributes, trained from scratch exactly as in Gauch et al.. It is the strongest label-only method on CAMELS. That paper reports a median NSE of 0.74 over 531 basins with 9 training years. It also reports that 53 basins suffice to match the calibrated mHM process model at about 0.66. We reproduce its basin fractions (2.5% to 100%) and record lengths (3, 6, 9 years, plus 1 year), so our label-scarcity curves sit on its axes. Beating it means a higher median NSE at the 2.5% and 10% basin fractions with 1 to 3 years of records, beyond the spread across five seeds.
XGBoost on lagged features is included because that paper found it ties EA-LSTM at 13 basins and 3 years, the regime we target. We give it the same EMA bands as features so it is not handicapped by long look-back windows.
The strongest existing pretrained method is a StefaLand-style masked autoencoder (Structured Physical Attributes Enable Efficient Foundation Models for Land-Surface Prediction). It is reimplemented with identical architecture, corpus, masking schedule, decoder, pretraining GPU-hours and fine-tuning budget, and differs from our model only in loss. The reported result we can anchor to is the 20% RMSE reduction over a supervised LSTM on ungauged basins. No NSE value for StefaLand on CAMELS-US was found on Lacuna, so our reimplementation is the operative comparison. This uniform-loss twin is also the key ablation: any gain it lacks is attributable to band weighting.
Two further ablations isolate the remaining ingredients. One feeds the causal bands as inputs to a supervised EA-LSTM with no pretraining. The other replaces the causal EMA targets with a non-causal VMD decomposition of the forcings, which tests whether causality matters for pretraining at all.
AutoCon pretraining applied to the same encoder is a secondary baseline. No reported result on any hydrology benchmark was found on Lacuna for AutoCon.
All models get the same hyperparameter search budget, 40 trials on a validation split of training basins. They also share the same test basins and the same five seeds.
Why not just…?
Why not just use plain masked pretraining as in StefaLand? This is the most serious alternative, and the uniform-loss twin tests it directly. If band weighting gains nothing over it at every label budget, our central claim is false and we will say so. Our reason to expect a gap is that a uniform squared error on daily forcing is dominated by unpredictable storms, so it spends most of its capacity on a signal that carries little information about storage.
Why not just give a supervised LSTM the multi-scale features? An LSTM can, in principle, learn exponential moving averages internally, and with 531 basins it probably does. That makes this a real threat at high label budgets. Our bet is narrower: with 13 to 50 basins and 1 to 3 years of records, the network cannot learn these filters from labels. The features-only ablation settles this. If it closes the gap, the contribution reduces to feature engineering, and we would report that.
Why not just decompose the flow record, as hybrid models do? It cannot run without a flow record, and its reported gains likely include leakage. We run a leakage audit that re-implements EEMD-LSTM (Daily Runoff Forecasting Using Ensemble Empirical Mode Decomposition and Long Short-Term Memory) on gauged CAMELS basins two ways: decomposing the full record once, and decomposing strictly online from past data only. If the online variant keeps a large margin over a plain LSTM, frequency handling on the target deserves a place in our pipeline, and we would add it for gauged basins.
Why not just use XGBoost when data are tiny? Gauch et al. show it ties deep models at the smallest budgets and caps out near NSE 0.66 with full data. We include it with the same bands. If it matches our model at the 2.5% basin fraction, deep pretraining is not justified in that corner.
Why not just use AutoCon's contrastive pretraining? Its positive pairs come from linear autocorrelation across fixed lags. Its authors report that this fails when cycles drift, which is exactly what our climate-shift split creates. The secondary baseline measures this directly.
Proposed Experiments and Decisive Checks
The main benchmark is daily CAMELS-US, under two test regimes.
The first is prediction in ungauged basins (PUB): k-fold cross-validation over basins, with held-out basins contributing forcing only during pretraining.
The second targets non-stationarity with a differential split-sample test. Models are fine-tuned on each basin's wettest years and tested on its driest years, and the reverse, which mimics the shifting regimes described in Identifying and Interpreting Hydrological Model Structural Nonstationarity Using the Bayesian Model Averaging Method.
Metrics are median NSE, NSE on log flows (to expose low-flow and drought skill), and peak-timing error in days (to expose the phase lag that decomposition papers complain about). A secondary check runs on hourly CAMELS-US over its standard 1990–2003 and 2003–2014 split. There the reference numbers are h-Diffusion at median NSE 0.774 and MTS-LSTM at 0.763 (Diffusion-Based Probabilistic Modeling for Hourly Streamflow Prediction and Assimilation). The question is whether storage-band pretraining narrows the gap in flashy basins without an external daily model.
Three outcomes would be decisive.
First, the proposal succeeds only if the band-weighted model beats its uniform-loss twin on median NSE in the scarce corner (10% of basins or fewer, 3 years or fewer). It must do so in both the ungauged and the dry-year tests, with the advantage shrinking toward zero at full data. That shrinkage pattern is the signature of a label-efficiency effect rather than a generally better model.
Second, the log-flow NSE gain should exceed the gain in ordinary NSE, consistent with the ECG finding that self-supervision helps most where labels are rarest.
Third, if the features-only supervised ablation or band-equipped XGBoost matches the pretrained model in that corner, we conclude that the storage bands, not pretraining, carry the benefit.
All of these results would be reported, along with the leakage audit, which by itself would tell the decomposition-hybrid community how much of its reported margin survives honest evaluation.
References
- Daily Runoff Forecasting Using Ensemble Empirical Mode Decomposition and Long Short-Term Memory
- Application of an improved LSTM model based on FECA and CEEMDAN VMD decomposition in water quality prediction
- A Data Scientist's Guide to Streamflow Prediction
- An improved hybrid data-driven model and its application in daily rainfall-runoff simulation
- Scaling Wearable Foundation Models
- Structured Physical Attributes Enable Efficient Foundation Models for Land-Surface Prediction
- Self-Supervised Contrastive Learning for Long-term Forecasting
- Self-Supervised Contrastive Forecasting
- Self-supervised representation learning from 12-lead ECG data
- The Proper Care and Feeding of CAMELS: How Limited Training Data Affects Streamflow Prediction
- Global prediction of extreme floods in ungauged watersheds
- CauSTream: Causal Spatio-Temporal Representation Learning for Streamflow Forecasting
- Diffusion-Based Probabilistic Modeling for Hourly Streamflow Prediction and Assimilation
- Identifying and Interpreting Hydrological Model Structural Nonstationarity Using the Bayesian Model Averaging Method