Background Leakage in Fixed-Camera Benchmarks

A camera trap, a traffic monitor, or a security feed looks at the same scene for months or years. When images from such a camera are shuffled at random into training and test sets, the test images share the exact rocks, branches, lighting, and often the same burst of near-identical frames with images the model trained on. The model can then score well by recognizing when and where a picture was taken, not what animal is in it. This proposal argues that the background is a temporal leakage channel, a way for future information to slip into training, that has not been measured. It also argues that the remedy is to model the background explicitly, using only past, empty frames from each camera.

We make two contributions. The first is a measurement. On iWildCam, we break the drop between a random split and a chronological (time-ordered) split into three parts: near-duplicate leakage, same-period background leakage, and genuine drift such as seasons and vegetation change. We use a ladder of increasingly strict splits for this, plus a test that swaps in backgrounds from other dates. The second is a method. A lightweight adapter on a frozen pretrained image model is trained to push its features away from anti-templates, meaning feature prototypes built from the camera's own recent empty frames. Those prototypes are refreshed using only data from before the prediction time. We ask whether this closes the drift gap without reopening the leak, and whether a frozen backbone can represent seasonal backgrounds well enough for this to work at all.

Core Claim

On fixed-camera data, much of the advantage a model shows under random splitting comes from the background being shared across the split boundary, not from better recognition of the target. We claim this share can be measured directly. We also claim that causal background anti-templates, built only from past empty frames at the same camera, recover more forward-in-time accuracy than three simpler options: cropping the animal out, plain fine-tuning, or prompt-based domain adaptation. A corollary concerns staleness. Anti-templates frozen at the training cutoff should lose value as the test date moves further from the cutoff, and could eventually start suppressing real foreground. The causal refresh is therefore the key ingredient, not background modeling as such.

Where This Sits

Three lines of prior work motivate the question. The iWildCam 2020 Competition Dataset documents that models latch onto static vegetation and lighting, and that fixed locations change with the seasons. WILDS turns that observation into a held-out-location benchmark and shows that robust training objectives rarely beat ordinary training. Neither work holds the camera fixed and splits by time, which is the realistic setting when a monitoring project keeps running the same cameras.

The temporal evaluation literature supplies the method of measurement. Evaluating Model Performance in Medical Datasets Over Time uses rolling backtests, where a model is trained on everything before a cutoff, tested just after it, and the cutoff is then moved forward. It finds random splits systematically optimistic. Reframing in Frequent Pattern Mining reports F-measure falling from 95% to about 70% when moving from random to sequential splits. That paper also names an open problem: such a drop mixes removed leakage with real drift. Temporal Convolutional Networks for the Advance Prediction of ENSO motivates leaving a buffer gap between training and test periods. All of this work is tabular or symbolic. None of it identifies an image background as the carrier of leakage. Our split ladder is designed to answer the mixing problem for images.

Background-aware learning supplies the mechanism. Context-Aware Correlation Filter Tracking adds a penalty on the filter's response to nearby background patches. Fine-grained Background Representation and LPCAM cluster background features into several prototypes instead of treating background as one class. DLST keeps both a first-frame template and a recent-frame template so that background suppression can follow drift. We reuse the context-penalty form and the multi-prototype idea. We apply them across months rather than across frames, and we use a source of negatives that tracking lacks: camera traps record large numbers of empty frames, which are pure background with no annotation needed.

Task-Disruptive Background Suppression shows that when the support background differs from the query background, background modeling adds noise. That finding cuts against our proposal: stale anti-templates could hurt. We test it directly as the staleness ablation.

Prompt-based adaptation on frozen backbones is the strongest competitor and also the source of the rigidity question. VDPG generates domain prompts for a frozen CLIP from a few unlabeled target images and separates domain context from object identity. It treats domains as unordered categories and keeps no spatial background model. Prompting-based Efficient Temporal Domain Generalization extrapolates prompts along time. It does so only for tabular data, and it assumes the frozen backbone can represent what lies ahead. Few-shot Adaptation of Medical Vision-Language Models is a cautionary result: elaborate prompt methods often fail to beat a linear probe. A linear probe is therefore a mandatory baseline here.

Research Program

The split ladder is the measurement core. Each camera's images are ordered by capture timestamp, which iWildCam records. We first confirm that the timestamps are reliable per camera and drop cameras whose clocks are inconsistent. Every rung uses the same cameras and the same number of training images. The rungs, from loosest to strictest, are:

  • Image-level random split. Images are shuffled individually.
  • Burst-grouped random split. Images from the same trigger burst stay on the same side.
  • Day-blocked random split. Whole days are assigned at random to train or test.
  • Chronological split with a buffer. Training uses data before a cutoff and testing uses data after a gap of several weeks.

The drop from each rung to the next is attributed to one leakage channel: near-duplicates, then shared background within a day or week, then drift across seasons. The last rung is repeated at several cutoffs in the style of a rolling backtest, which yields an accuracy-versus-time-since-cutoff curve.

A background-swap probe gives a second, independent estimate. For each test animal we take the detector's box and paste the animal onto an empty frame from the same camera. In one condition the empty frame comes from a date inside the training period; in the other it comes from a date after the cutoff. The difference in accuracy between the two conditions isolates how much the model relies on recognizing the background's date. A second probe asks whether a frozen backbone's features encode season: a linear probe on empty-frame features predicts month. High accuracy would mean the backbone already represents background drift, so an adapter can work with it.

The method is a small adapter, plus a classifier head, on a frozen CLIP image encoder. The adapter is trained with the usual species loss plus a context penalty in the spirit of Müller et al. We write $\hat{y}$ for the predicted species, $y$ for the label, $\phi(x)$ for the adapted feature of image $x$, $c$ for the image's camera, and $t$ for its capture time. $\{b_k^{c,t}\}$ are $K$ prototypes obtained by clustering the features of camera $c$'s empty frames captured in the window $[t-W, t)$. The loss is

$$ \mathcal{L} = \mathrm{CE}(\hat{y}, y) + \lambda \sum_{k=1}^{K} \max\!\big(0,\ \cos(\phi(x), b_k^{c,t}) - m\big), $$

where $\lambda$ weights the penalty and $m$ is a margin. At test time the prototypes are rebuilt from empty frames available before the test image. This requires no labels, because the empty-versus-animal flag can come from an off-the-shelf detector. Because prototypes are only ever built from the past, the method cannot leak future information by construction.

If the season probe shows the frozen features cannot tell seasons apart, we unfreeze only the last two blocks of the backbone. That gives a controlled test of whether representation rigidity, not the objective, is the bottleneck.

Baselines

The naive reference point is ordinary fine-tuning of the full model with standard augmentation, trained on the pre-cutoff data. It is evaluated on every rung of the ladder. Its own drop from the random rung to the chronological rung is the leakage-plus-drift gap that every other method must be read against.

The frozen-backbone reference is a linear probe on CLIP features. Shakeri et al. show this simple baseline often matches prompt methods, so beating it is the minimum bar for any adapter.

The strongest existing method is VDPG, which reports 46.5% Macro F1 on WILDS iWildCam against 38.3% for earlier methods (VDPG). We run it with its few-shot domain images drawn only from each camera's pre-test window, so it receives the same unlabeled data our anti-templates use. It gets the same backbone and the same hyperparameter-search budget (an equal number of trials on a pre-cutoff validation block). That reported figure is on the held-out-location split. We found no reported result on Lacuna for any method under a within-camera chronological split of iWildCam, so all numbers in our protocol will be new.

The 2020 competition baseline, an Inception-v3 with a class-balanced loss, reached 0.62 macro F1 on that dataset's own test split (The iWildCam 2020 Competition Dataset). We cite it for context only, since it is a different split.

The decisive ablations isolate the causal refresh. We compare four variants of where the prototypes come from:

  • Frozen at the cutoff. Prototypes are built once from pre-cutoff empty frames and never updated.
  • Leaky. Prototypes are built from empty frames that include future dates.
  • Random negatives. Background patches are drawn at random instead of from empty frames.
  • No penalty. Prototypes are dropped ($\lambda = 0$).

Our method beats the field only if all of the following hold. It must improve chronological-rung Macro F1 over VDPG and over full fine-tuning, with the gain larger than the spread across seeds and cutoffs. It must widen its advantage over the frozen-at-cutoff variant as time since the cutoff grows. And its date dependence, as measured by the background-swap probe, must be smaller.

Why not just…?

Why not just crop the animal with a detector and classify the crop? This is the most serious threat, because cropping removes most of the background at once. We found no reported result on Lacuna for crop-based classification under chronological splits. We will run it as a full baseline on every rung. Crops still contain a ring of local background around the animal, and the swap probe will show whether crop classifiers remain date-sensitive. If crops close the chronological gap as well as anti-templates do, we will report background modeling as unnecessary for this task. The split ladder and the probes still stand as contributions in that case.

Why not just hold out whole cameras, as WILDS does? Holding out cameras answers a different deployment question: how well does the model do at new sites? The realistic case for ongoing monitoring projects is future dates at known sites. Both Zhou et al. and Beery et al. indicate that fixed sites still drift over time. We report the location split alongside ours, so readers can see whether the two splits rank methods differently.

Why not just subtract a per-camera median background image? This classical pixel-level background subtraction is cheap. It is also known to break down under lighting changes and moving vegetation, the very conditions seasonal drift produces. We include it, computed causally with the same window $W$. If it matches our feature-space anti-templates, then the frozen backbone adds nothing and the method should be simplified.

Why not just fine-tune the whole model or use a larger backbone? The camera-trap study by Deep learning-based ecological analysis of camera trap images is impacted by training data quality and quantity found that moving from ResNet18 to ConvNeXt-T barely changed downstream ecological accuracy. That suggests scale alone does not fix a problem in the data. We still run full fine-tuning as the naive baseline. Our partial-unfreezing condition tests directly whether representational capacity is what limits performance.

Decisive Checks

The study succeeds as a measurement if two independent estimates agree: the share of the random-versus-chronological gap attributed to background by the ladder, and the share attributed by the swap probe. Their agreement should hold across several cutoffs.

It succeeds as a method under four conditions. Causal anti-templates must beat VDPG, the linear probe, detector crops, and median subtraction on chronological Macro F1. The frozen-at-cutoff variant must decay with time since the cutoff while the refreshed one does not. And the leaky variant must look best on random splits but not on chronological ones. That last pattern would directly show the background acting as a leakage channel.

If the frozen backbone cannot predict month from empty frames, and partial unfreezing is what restores the gain, the conclusion changes. The result would then be evidence that representation rigidity, not the training objective, limits background suppression across long time spans.