Solar power drops when a cloud crosses the sun, and the forecasts grid operators rely on are worst exactly then. Deep forecasters trained to minimize squared error learn to predict smooth averages. They lag behind sudden drops and rises in output ("ramps") and underestimate extremes. Meanwhile, the geostationary satellite imagery that shows clouds approaching is abundant and unlabeled, but current models use it only through supervised training on a handful of ground irradiance sensors. We propose to pretrain a satellite-plus-station forecaster without labels. The pretraining task is to predict the internal features of future frames, rather than their raw pixels. A slowly updated copy of the network (the "teacher") sees the whole sequence, including the future. The learning network (the "student") sees only the past and must predict what the teacher's features will be for the frames it cannot see.
The bet is specific. When you predict future cloud images pixel by pixel, the prediction blurs over the many places a cloud edge might end up. Predicting abstract features should instead average over the possible cloud states, such as "a front arrives in 40 minutes" versus "it dissipates". That kind of state is what determines a ramp. If the bet holds, the gain should show up mainly on ramp events and in low-label, unseen-station settings, not just as a small shift in average error. The experiments are designed so that a null result on ramps kills the idea.
Core Claim
Masked prediction of contextualized latent targets has been shown to learn semantics faster and better than pixel reconstruction in vision, speech and text. The key sources are data2vec 2.0, which regresses a teacher's averaged multi-layer features and matches MAE 16.4x faster, and BootMAE, which uses a moving-average teacher and reaches 84.2% ImageNet top-1. We claim that the same objective, applied to spatiotemporal satellite sequences with the masked region placed on future time blocks, yields a pretrained forecaster that improves ramp-conditioned skill and label efficiency over three alternatives:
- pixel-reconstruction pretraining on identical data;
- supervised multi-site pretraining;
- purely supervised end-to-end training.
We make no claim that it reduces average error across all conditions. That may or may not follow.
The objective is simple to state. Let $x_{\le t}$ be past satellite frames and station history. Let $f_\theta$ be the student encoder and predictor, and $g_{\bar\theta}$ the teacher, whose weights $\bar\theta$ are a moving average of $\theta$. For each future token position $j$ in the horizon $(t, t+H]$, the target $y_j$ is the average of the teacher's top $K$ layer outputs at position $j$, each instance-normalized, computed from the full sequence $x_{\le t+H}$. The loss is
$$\mathcal{L}(\theta) = \sum_{j \in (t,\,t+H]} \big\| f_\theta(x_{\le t})_j - y_j \big\|_2^2 .$$
The instance-normalized multi-layer targets follow data2vec 2.0, whose recipe is our main defense against the collapse and drift risk noted for bootstrapped regressors. Because the teacher sees the future, its features encode what actually happened in the context of the whole trajectory. The student must therefore learn which present configurations lead to which future cloud states.
Relation to Prior Work
Four lines of work bracket this idea.
Supervised forecasters that already fuse imagery and time series. CrossViViT mixes EUMETSAT satellite cubes and station series with cross-attention and quantile heads. SIAT rolls a GPT-2 decoder forward over ViT latents of sky images. SMT fuses a webcam panorama with 24 hours of GHI history. Baguan-solar predicts cloud fields in pixel space from Himawari imagery plus a weather foundation model, then maps them to irradiance.
All four learn their representations only from irradiance labels, or, in Baguan-solar's first stage, from explicit pixel-level cloud fields. SIAT comes closest because it forecasts in latent space. But its latents are defined by regression on current irradiance, which caps what they can encode at "how bright is it now". It also reports trouble with peaks above 900 W/m². Our latents are defined without labels by a teacher that has seen the future. These papers motivate the proposal, and CrossViViT's released code supplies the architecture we build on.
Dense auxiliary targets for ramps. ECLIPSE shows that adding a decoder that predicts future sky segmentation (sun, cloud, clear) cuts temporal distortion at 2 minutes from 11.3% to 9.2% and reduces 95th-percentile errors. This is our most important piece of motivating evidence: dense predictive targets about future cloud state help with ramps. ECLIPSE needs segmentation masks, however. We replace them with self-supervised feature targets, the move that MVP and MaskDistill made in vision when they replaced tokenizers and labels with continuous teacher features.
Constraining evidence. Two results constrain the design. UniLiP observes that semantic feature targets discard fine spatial detail. For solar, fine cloud edges may be the signal, so the latent-versus-pixel comparison is a genuine open question, not a formality. VisionTS shows that a frozen ImageNet pixel MAE can forecast generic series zero-shot, which raises the possibility that generic image priors are enough. Both become explicit ablations.
Label-efficient transfer. Paletta et al. show that supervised multi-site pretraining cuts local data needs by 80%, and that off-site failure comes mostly from differences in measurement scale rather than in what the images look like. This sets the bar our label-free pretraining must clear in the low-data regime.
Research Program
Base setup. We build on the MIT-licensed CrossViViT repository and its BSRN ground stations plus EUMETSAT Rapid Scan pipeline, keeping its satellite and station encoders and cross-attention mixer. The satellite stream uses visible, infrared and water-vapor channels. The infrared and water-vapor channels stay informative at night, so pretraining runs around the clock on unlabeled imagery. This gives a direct handle on the dawn and dusk discontinuity that Baguan-solar solved with a two-stage design. Station tokens are clear-sky-normalized, as in SMT and Baguan-solar.
Stage 1: label-free pretraining. We pretrain on the full 2008–2022 satellite archive, far more frames than have paired labels. Masking follows data2vec 2.0's inverse block scheme but along time: contiguous future blocks are hidden, plus a small fraction of random spatial blocks in the past to keep spatial context learning.
Three components come directly from data2vec 2.0:
- the student encodes only visible tokens;
- a shallow convolutional predictor fills in the masked positions;
- one teacher pass is reused across several masked views of the same sequence ("multi-masking").
Multi-masking matters here because CrossViViT reports roughly 5 hours per epoch, and this reuse of the teacher pass is what made data2vec 2.0 cheap.
Stage 2: fine-tuning. We fine-tune with CrossViViT's multi-quantile head on station GHI. The downstream quantiles are then usable for risk-aware dispatch of the kind Dall'Anese et al. formulate with Conditional Value-at-Risk. Optimizing dispatch itself is out of scope.
Analysis. We analyze what the pretrained latents encode. Linear probes will test whether frozen features predict the timing of ramp onset at a station, and whether they predict cloud-edge distance from the station. If the latents do not linearly separate "front arriving" from "stable overcast" better than pixel-MAE latents do, the proposed mechanism is wrong even if the aggregate numbers improve.
Baselines
Naive reference. The naive reference is smart persistence: the current clear-sky index projected forward onto the clear-sky curve. It needs no training and is the standard skill denominator. Every model is reported as forecast skill relative to it on the same test days.
Strongest existing method. The strongest existing method on our primary benchmark is supervised CrossViViT itself. It fuses satellite and station inputs and produces calibrated quantiles, which is exactly our task. The number to beat is its reported MAE of 50.35 at the Cabauw (CAB) station, against 54.91 for the best time-series baseline (CrossViViT). The file gives the figure without units. It also reports test confidence above 0.90 at unseen stations for a 96% interval.
We rerun it from the official code on our splits rather than trusting the reported figure. We then give it the same total GPU budget our method spends on pretraining plus fine-tuning. Concretely, the supervised model trains for more epochs until the compute matches, with an equal hyperparameter search budget. Beating it means two things. On CAB and on the held-out stations, we must improve ramp-conditioned MAE and 95th-percentile error by a margin larger than the seed-to-seed spread across three seeds. We must also not lose on overall MAE or interval coverage.
Pixel-MAE ablation. This ablation isolates our key ingredient. The same architecture, data, masking and compute are used, but the predictor regresses masked future pixels instead of teacher features. It is the most dangerous baseline, and the proposal stands or falls on beating it on ramps.
Masking ablation. A second ablation keeps latent targets but uses random spatial masking with no future blocks. This tells us whether the forecasting-shaped mask, rather than latent targets alone, carries the effect.
Supervised multi-site pretraining. This baseline follows Paletta et al.: pretrain on all labeled stations, then fine-tune on the target station. It is compared across label fractions.
Generic-feature teacher. Here the student predicts frozen ImageNet-pretrained ViT features at masked positions, in the style of MaskDistill. It tests whether the teacher must learn from atmospheric data at all.
Simple statistical baseline. Clear-sky-detrended autoregression, following the ensemble-deduction AR model, reportedly beats deep nets on single-site series.
Secondary nowcasting benchmark. On SIRTA sky-camera nowcasting, the reference points are ECLIPSE (forecast skill 24.0% at 10 minutes; temporal distortion 9.2% at 2 minutes) and SIAT (RMSE 76.94 W/m² against persistence at 93.33 W/m²).
Why not just…?
Why not just pretrain a pixel masked autoencoder on the same satellite archive? This is the real threat. UniLiP's observation that feature targets lose fine geometry suggests pixel targets might preserve cloud-edge detail that matters for timing. We do not argue it away. The pixel-MAE ablation is matched on data, architecture, masking and compute, and evaluated on ramp-conditioned error, temporal distortion and the linear ramp-onset probe.
If pixel MAE ties or wins on ramps, the conclusion is that pretraining on the unlabeled archive is what helps, not the target space, and we would report that. There is one evidence-based reason to expect latents to win. In vision, pixel regression spends capacity on texture (BootMAE had to route shallow features to a separate pixel decoder to free the encoder). For future frames, pixel squared error additionally rewards blur over uncertain cloud positions.
Why not just do supervised pretraining across many stations, or train CrossViViT longer? Paletta et al. show supervised multi-site pretraining is strong, with an 80% cut in local data. But it is limited to frames that have ground labels, and it inherits the station-mix skew that gave CrossViViT its clear-sky bias. The compute-matched CrossViViT run answers "train longer". The label-fraction curves, at 5%, 20% and 100% of target-station labels, answer "supervised pretraining". Label-free pretraining should matter most at the low-label end. If the curves coincide there, the unlabeled archive adds nothing.
Why not just use a frozen generic model such as VisionTS on the station series? VisionTS ranks first on GIFT-Eval zero-shot, but it only sees the 1D series folded into an image. It cannot see an approaching cloud front, which CrossViViT's gain over time-series baselines (50.35 versus 54.91 MAE) shows is worth having. We run VisionTS zero-shot and with its one-epoch normalization-layer fine-tune as a series-only reference. The generic-feature-teacher ablation separately tests whether generic image semantics suffice when imagery is available.
Why not just detrend with a clear-sky model and fit an autoregression? For a single site in stable climates this can match deep models cheaply. But autoregression extrapolates from the station's own past, so by construction it cannot anticipate a ramp caused by a cloud arriving from elsewhere. We include it so that any claimed ramp gain is measured against the cheapest competent forecaster.
Proposed Experiments and Decisive Checks
Primary benchmark. The primary benchmark is BSRN plus EUMETSAT day-ahead forecasting with CrossViViT's station splits, including its held-out stations. We report MAE, RMSE and pinball loss, 96% interval coverage, and forecast skill against smart persistence.
Ramp-conditioned metrics. Most importantly, we report ramp-conditioned metrics. We define ramp windows as changes in clear-sky index above a fixed threshold within 30 minutes. On those windows we report error, the temporal distortion index used by ECLIPSE, and 95th-percentile error. We also report dawn and dusk windows separately, to test whether around-the-clock infrared pretraining removes the edge-of-day instability.
Secondary benchmark. A secondary run adapts the pretraining to SIRTA sky-camera sequences. There we test whether label-free latent targets recover ECLIPSE's segmentation-driven ramp benefit without masks. We compare temporal distortion at 2 minutes and skill at 10 minutes against its reported 9.2% and 24.0%. Chilbolton serves as a harder maritime check against SIAT's 21.45% skill.
Decisive outcomes. The central check is the three-way comparison of latent-future pretraining against pixel-MAE pretraining and against compute-matched supervised CrossViViT, restricted to ramp windows at held-out stations.
- Support. Latent targets beat pixel targets on ramp error and temporal distortion by more than seed variance, the linear probe shows better ramp-onset decoding, and overall MAE is no worse than CrossViViT's rerun.
- Partial support. Both pretraining variants beat supervised training but tie each other. The value then lies in the unlabeled archive, not in semantic targets.
- Failure. Neither beats compute-matched supervised training on ramps. Label-free pretraining on satellite sequences then does not address intermittency, whatever it does to average error.
References
- Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language
- Bootstrapped Masked Autoencoders for Vision BERT Pretraining
- Improving *day-ahead* Solar Irradiance Time Series Forecasting by Leveraging Spatio-Temporal Context
- Solar Irradiance Anticipative Transformer
- Solar Multimodal Transformer: Intraday Solar Irradiance Predictor Using Public Cameras and Time Series
- Integrating Weather Foundation Model and Satellite to Enable Fine-Grained Solar Irradiance Forecasting
- ECLIPSE : Envisioning Cloud Induced Perturbations in Solar Energy
- MVP: Multimodality-Guided Visual Pre-training
- A Unified View of Masked Image Modeling
- UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing
- VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters
- Sky-image-based solar forecasting using deep learning with multi-location data: training models locally, globally or via transfer learning?
- Optimal Dispatch of Residential Photovoltaic Inverters Under Forecasting Uncertainties
- Modified Auto Regressive Technique for Univariate Time Series Prediction of Solar Irradiance