Swapping Seasons to Find Real Change in Satellite Image Time Series

Satellites photograph the same place every few days. Most of what differs between two photos of one location is not change at all: snow arrives and melts, crops green up and are harvested, the sun drops lower, and shadows lengthen. Real change, such as a new building, a cleared forest, or a flooded road, is usually small and buried under these seasonal differences. Current self-supervised methods respond by training features that ignore time. That makes the features robust to seasons, but it also makes them partly blind to the changes we care about. We propose a different approach. Each image is split into a spatial content map (what is physically there) and a dense, low-capacity style map (how the season and lighting currently render it). The model is then asked to re-render one date's photo using another date's content. Wherever that swap fails to reproduce the real photo, something physical changed.

In two sentences: we learn a decoder that can paint any date's seasonal appearance onto any other date's scene structure. Where the swapped rendering disagrees with the actual photo, we have found real change rather than a season. That disagreement map plays two roles. During pre-training it decides, pixel by pixel, which locations should be held constant across time. After pre-training it serves as a change detector that needs no labels.

Core Claim

Content-style disentanglement is useful for change detection only if it can tell seasonal style apart from permanent change. Splitting an image into two codes does not do this by itself. If the content code is forced to be identical across dates, real change gets pushed into the style code, and the model is change-blind again, just with a different address for the blindness.

Our claim is that a re-rendering residual resolves this. The residual is computed per pixel, and the model generates it for itself during training. It can then serve as a soft mask on the content-invariance loss. With that mask, content stays invariant only where the scene is physically static. We further claim that this works only when two conditions hold. The style representation must be dense but spatially coarse, so that it can describe snow on one parcel and bare soil on the next without being able to encode building outlines. And training must be uncooperative, so that the decoder cannot smuggle structure through the style channel.

Let $c(x)$ be the content map of image $x$ at high spatial resolution. Let $s(x)$ be its style map: per-window channel means and variances at about 1/16 resolution. Let $D$ be a decoder. For two dates $t$ and $t'$ at the same location, the swapped rendering and its per-pixel residual are

$$\hat{x}_{t} = D\big(c(x_{t'}),\, s(x_t)\big), \qquad r_p = \lVert x_{t,p} - \hat{x}_{t,p} \rVert,$$

where $p$ indexes pixels. The content-invariance term is weighted by $w_p = \sigma\big((\tau - r_p)/\beta\big)$. Here $\sigma$ is the logistic function, $\tau$ is a threshold set from a per-batch residual quantile, and $\beta$ is a temperature. The term is

$$\mathcal{L}_{\text{inv}} = \sum_p w_p \,\lVert c_p(x_t) - c_p(x_{t'}) \rVert^2 .$$

The full objective adds three more terms: a same-date reconstruction loss, a contrastive term that makes style maps of the same season in different years agree, and a content contrastive term against other locations. Without that last term, $c$ can collapse to a constant.

Relation to Prior Work

Three methods motivate this proposal by exposing the gap. Seasonal Contrast treats co-located images from different seasons as positive pairs, with separate heads meant to be either seasonally invariant or variant. However, nothing in the architecture stops those heads from sharing entangled features, and true changes inside the sampling window are pulled together like seasons. Change-Aware Sampling and Contrastive Learning (CACo) fixes the worst of this. It compares long-gap and short-gap feature distances and fits a mixture model to decide whether a whole image pair is positive or negative. That decision is per image and binary, so a tile with one new building among fields is either entirely pulled together or entirely pushed apart. Our mask makes the decision per pixel, and it comes from an explicit model of seasonal appearance rather than a ratio of distances. Semantic Decoupled Representation Learning attacks the related problem of background dominance with foreground/background masked pooling. It needs building masks, and it separates spatial classes rather than separating appearance from structure. We need no masks.

Three methods supply the mechanism. From Image Disentanglement and Uncooperative Re-Entanglement we take the rule that a network is updated only when its input is real. In our setting, the encoders are frozen when they re-encode swapped renderings for a cycle loss. That paper reports 0.998 versus 0.695 correlation with true factors on synthetic data when cooperative cheating is removed. From ALADIN we take channel mean-and-variance statistics as a style carrier that holds no coordinate information. From DSI2I we take the evidence that a single global style vector bleeds across regions, so style must be dense. We compute ALADIN-style statistics inside coarse windows, a compromise between those two papers. We do not need DSI2I's optimal-transport warping, because co-located satellite images are already geometrically aligned.

Two lines of work constrain the design. Frequency-splitting methods, including the Gaussian Fourier swap in Basak and Yin, assume style is low-frequency. Snow lines and cast shadows are not, which is why our style is learned rather than defined by a filter. PISCO shows that linear post-hoc projection can strip style when pairs with the same content and different styles are available. As discussed below, multi-temporal imagery supplies such pairs for free, so PISCO is a real competitor rather than a strawman. Finally, Self-supervised Pre-training Enhances Change Detection in Sentinel-2 Imagery finds that intermediate layers localize change best. This finding motivates keeping the content map at high resolution.

Research Program

We pre-train a ResNet-50 encoder on Sentinel-2 revisit imagery that contains both short gaps (months) and long gaps (years). This follows the CACo sampling protocol over the SeCo city locations. A second run uses the phenology-aligned SSL4Eco sampling, to test whether biome-diverse seasons make style learning harder or easier.

Content is taken from an intermediate stage at 1/4 resolution. Style is taken from all stages as window statistics at 1/16 resolution. During training, style windows are randomly shuffled within each tile, following DSI2I, so that style cannot carry layout.

The analysis has three parts. The first is a leakage audit: linear probes on content should be unable to predict the season or the phenological phase, while probes on style should predict phase well. The second is a residual audit: the residual should be high on labeled change pixels and low on labeled no-change pixels across seasons. The third measures how the residual-weighted invariance changes the learned features compared with all-pixel invariance.

Baselines

The naive reference points are ImageNet initialization and plain change vector analysis on raw bands. On OSCD, Seasonal Contrast reports F1 36.20 for ImageNet and 46.94 for SeCo. These two numbers anchor the label-supervised comparison.

The strongest existing self-supervised competitor is CACo. On OSCD it reports an 8.5% relative F1 improvement over the best baseline, and 93.08% accuracy on EuroSAT with ResNet-18. The file gives no absolute OSCD F1 for CACo, so we will re-run it. It is the strongest competitor because it already addresses change blindness, on the same data, with the same backbone family. We will retrain CACo and our method on identical locations, dates, and image counts, with the same backbone, epochs, and hyperparameter search budget. Our method counts as beating CACo only if it improves OSCD F1 and DynamicEarthNet mIoU beyond seed variance (three seeds each) while matching CACo within one point on EuroSAT and BigEarthNet. Matching on the static tasks matters, because winning change detection by sacrificing static semantics would not be progress.

For the low-label regime we compare against SDRL on LEVIR-CD with 1% of the labels, where SDRL reports F1 46.62 and SeCo 40.78. SDRL uses masks; we do not. Matching it would therefore be meaningful, and exceeding it would be strong. For label-free change detection, the reference is pretext-trained features plus change vector analysis. Self-supervised Pre-training Enhances Change Detection reports OSCD F1 45.79 for that setup. We compare it against our raw re-rendering residual, thresholded the same way. As a supervised ceiling we fine-tune SARAS-Net heads on each encoder. That paper reports F1 91.91 on LEVIR-CD with full labels.

We isolate the key ingredient with five ablations. The first removes the residual mask, which gives all-pixel invariance and is effectively a dense SeCo. The second removes the style branch and masks with feature differences, which is effectively a per-pixel CACo. The third replaces dense style with one global style vector. The fourth uses cooperative training. The fifth replaces learned style with a fixed low-frequency Fourier swap.

Why not just…?

Why not just difference CACo features? CACo's features may already be change-sensitive enough that change vector analysis on them matches our residual. Its pair decision is per image, so the prediction is that it degrades on tiles where change is a small fraction of area. We will test this directly by stratifying OSCD and LEVIR-CD tiles by changed-pixel fraction. If CACo matches us in the low-fraction stratum, the per-pixel mask is not earning its cost, and we would report that.

Why not just project out style post-hoc with PISCO, using seasonal pairs as the style manipulations? This is the most serious threat. Same-location, different-season pairs are exactly the paired data that PISCO needs and otherwise lacks. Its known weakness is a single global linear subspace, and phenology varies by land cover. We will apply PISCO to CACo features using our training pairs. If it comes within seed variance of our method on OSCD and DynamicEarthNet, the conclusion is that linear removal suffices. We would then reframe the contribution around the label-free residual detector, which PISCO does not provide.

Why not just normalize radiometry or swap low frequencies before differencing? Fourier swapping assumes style is smooth. Basak and Yin note artifacts even with Gaussian masks, and snow lines and shadows violate the assumption. The Fourier ablation measures this with the same encoder. Its false-alarm rate on unchanged winter-versus-summer pairs is the decisive number.

Why not just train a supervised attention network? SARAS-Net is strong with dense labels, but our question is the regime where such labels are scarce. We report it as a ceiling and fine-tune its heads on every encoder. That way the comparison is between pre-training methods, not between architectures.

Decisive Checks

Three outcomes would falsify the proposal. The first is if content probes predict phenological phase about as well as style probes do, which would mean the factorization failed. The second is if the masked model does not beat the unmasked ablation on OSCD F1, which would mean the residual mask adds nothing. The third is if PISCO-on-CACo matches us. In the other direction, three results would confirm the central claim. The re-rendering residual should exceed 45.79 F1 on OSCD with no labels. The false-alarm rate on unchanged cross-season pairs should be lower than for both CACo-plus-change-vector-analysis and the Fourier swap. And the leakage audit should show that season information sits in style and not in content.