Foreground Clip Normalization for Small-Data Animal Video Re-Identification

Identifying individual animals from video instead of tags or collars depends on a network seeing the animal rather than its surroundings. In practice, models often take the shortcut. On WildlifeReID-10k, accuracy falls by up to 12.3% for MiewID once train and test images stop sharing near-identical backgrounds (WildlifeReID-10k). On PolarBearVidID, an image model made 81 cross-zoo identity errors (PolarBearVidID). We propose a specific architectural fix. A small video network is built from factorized 3D convolutions, meaning each 3D filter is split into a per-frame 2D spatial filter followed by a 1D filter across time. One set of these filters is shared across several image resolutions. Between the spatial and temporal filters sits a normalization layer whose statistics are computed once per clip, over the animal's pixels only. That layer removes each clip's lighting and colour cast before the network mixes information across frames. It also absorbs the large statistical differences between the coarse and fine copies of the shared filters.

The idea fits in two sentences. Background, site and lighting cues enter a video model through its normalization layers, because those layers either average over the whole frame or apply one fixed rescaling learned from the training sites. If each clip is instead normalized using statistics of the animal alone, pooled over the whole clip so that nothing flickers frame to frame, a parameter-light shared-scale video network should transfer across sites better than one with standard normalization. We test that claim against the obvious cheaper fixes.

Core Claim

We claim that foreground-masked, clip-level instance normalization, placed between the spatial and temporal halves of each factorized block, reduces site leakage in animal video re-identification. The gain should come from removing within-scale variation across clips, which no static normalization can absorb. It should not come merely from repairing the coarse-versus-fine statistical gap that cross-scale weight sharing creates.

The distinction matters. A skeptical reviewer will point out that the scale gap is a discrete, known shift. Per-scale static normalization (a separate scale-and-shift for each pyramid level) can fix it for free, and it folds into the convolution weights at inference. We accept this and build it in. The proposal stands or falls on whether the instance-level component adds anything beyond per-scale static normalization once train and test come from different sites. If it does not, the result is still useful: it means weight-shared video Re-ID needs only foldable per-scale normalization.

Research Program

Backbone. The backbone is a compact residual network of factorized blocks: a $1\times3\times3$ spatial convolution, a normalization layer, a nonlinearity, then a $3\times1\times1$ temporal convolution.

Cross-scale sharing. Each clip is processed at two or three spatial resolutions by the same block weights, in the style of Scale-Recurrent Network for Deep Image Deblurring.

  • That paper shared one encoder-decoder across an image pyramid with under a third of the parameters of multi-branch models.
  • We drop its sequential recurrent memory and instead concatenate pooled embeddings across scales, so the scales run in parallel.
  • Here sharing serves as a regularizer. The datasets are tiny (13 polar bears), and we hypothesize, but have not shown, that splitting supervision across separate per-scale branches would overfit.

The normalization layer. Let $x_{c,t,h,w}$ be the feature at channel $c$, frame $t$ and spatial position $(h,w)$, computed at pyramid scale $s$. Let $m_{t,h,w}\in\{0,1\}$ be the animal mask, downsampled to that scale. The layer computes a mean $\mu_c$ and standard deviation $\sigma_c$ over all masked positions in the clip, then outputs

$$y_{c,t,h,w} = \big(\gamma_{s,c} + \alpha_c \tanh(w_c^\top z)\big)\,\frac{x_{c,t,h,w}-\mu_c}{\sigma_c+\epsilon} + \beta_{s,c}.$$

In this equation:

  • $\gamma_{s,c}$ and $\beta_{s,c}$ are static per-scale parameters.
  • $z$ is a descriptor of the clip, made of its pre-normalization foreground $\mu$ and $\sigma$ across channels.
  • $w_c$ is a learned projection, initialized to zero.
  • $\alpha_c$ is a learned gain.

The bounded, zero-initialized dynamic term follows SeeDNorm, which found unbounded dynamic scaling unstable and showed that normalization discards useful magnitude information. Feeding $z$ back in lets the network keep, for example, a clip's overall contrast as a cue if that cue helps. It is the same idea as Q-Norm, which generates the scale and shift from an image-quality feature, but without Q-Norm's separate quality-assessment network.

Why the statistics are pooled over the clip. Pooling over $t$ rather than computing them per frame avoids the frame-to-frame flicker known to affect per-frame instance normalization. It also means the temporal filter sees a consistently scaled sequence. Restricting the pool to foreground pixels is what separates this from ordinary instance normalization (Improved Texture Networks). Ordinary instance normalization lets snow, water and enclosure walls set the statistics.

Masks and training.

  • Masks come from the zero-shot DINOv2-LDA-SAM2 pipeline of RoVF. That paper reports 97.26% segmentation accuracy and 92.77% IoU on PolarBearVidID, and 73.14% IoU on the meerkat data.
  • Training uses cross-entropy plus hard-mined triplet loss, following Shang et al..
  • On meerkats we add RoVF's negatives: animals seen together in one frame are guaranteed to be different individuals.

Known limitation. We do not add frame alignment. Appearance-Preserving 3D Convolution showed that temporal filters smear texture when body parts move between frames. We treat this as a known confound and measure it with one ablation, rather than folding a second mechanism into the claim.

Baselines

Naive reference: per-frame image model. A ResNet50 embeds each frame and the embeddings are averaged over the clip.

  • Reported result: on PolarBearVidID, PolarBearVidID gives 85.3% ± 1.1% Rank-1 and 45.4% ± 2.4% mAP.
  • We rerun it on our splits with our masks, so the comparison is like for like.

Strongest video baseline on that dataset: GLTR. This is a temporal aggregation network over 2D frame features.

  • Reported result: 96.6% ± 1.6% Rank-1 and 88.2% ± 9.0% mAP on the dataset's own protocol (PolarBearVidID). The large mAP variance means differences of a few points will not be meaningful without many seeds.
  • Fair comparison: we retrain GLTR on our leave-one-zoo-out splits, with masked inputs and the same augmentation.
  • Beating GLTR means higher mean mAP on held-out zoos across at least five seeds. That is the regime where its near-ceiling random-protocol number is least informative.

Strongest low-supervision baseline: RoVF. This is a frozen DINOv2 backbone with a recurrent Perceiver.

  • Reported results: 55.0% Top-1 on masked PolarBearVidID and 46.5% on masked meerkat test sets (RoVF).
  • We use RoVF's own protocol and masks, so its numbers are directly comparable.
  • Beating RoVF means exceeding those figures with a model that has orders of magnitude fewer parameters. Matching them would already be informative.

Ablations that isolate the key ingredient. All ablations hold architecture, data, epochs and a fixed hyperparameter-search budget constant. The core is a two-by-two comparison:

Static Foreground clip statistics
Shared normalization across scales standard batch norm —
Per-scale normalization per-scale batch norm (the cheap fix) the proposal

Further variants:

  • whole-frame clip instance normalization (no mask);
  • per-frame foreground instance normalization (tests the flicker concern);
  • the proposal with $w_c$ fixed at zero (no dynamic term);
  • separate unshared per-scale branches at matched FLOPs.

Why not just…?

Why not just use per-scale batch normalization? This is the most serious threat. It repairs the scale gap at zero inference cost. Our 2×2 ablation settles the question: if per-scale batch norm matches the proposal on held-out zoos, the instance component is unnecessary. We would report exactly that. We expect the gap to appear only on the cross-site split, not the random split.

Why not just mask the background and add photometric augmentation? RoVF already masks inputs, and augmentation randomizes colour. Masking alone leaves the animal's own appearance under site lighting, such as snow glare and underwater tint. Augmentation must guess the right range of distortions. The masked-input, heavily augmented, static-normalization model is therefore a required baseline. A zoo-prediction probe on the embeddings shows which approach actually removes site information.

Why not just use a frozen foundation model? RoVF reports that fine-tuning DINOv2 on these small datasets overfit. Its frozen version is our baseline above. If it wins outright, the case rests on cost: Animal Re-Identification on Microcontrollers shows field hardware has 256 KB of SRAM. We report parameters and per-clip latency, but we do not claim microcontroller deployment.

Why not just use full 3D convolutions? Within a fixed budget, the factorized form costs fewer parameters (the 55.86% reduction above). It also gives the normalization layer a natural place to sit, between the spatial and temporal filters. A matched-parameter full-3D variant is included as an ablation.

Proposed Experiments

PolarBearVidID (primary). The decisive test uses leave-one-zoo-out splits: 13 bears across 6 zoos.

  • Metrics: Rank-1, mAP, the count of cross-zoo errors (the measure that fell from 81 to 6 in the original paper), and a linear probe's accuracy at predicting zoo from embeddings.
  • Success means two things together: lower probe accuracy and higher held-out mAP for the proposal than for per-scale batch norm and for the masked-and-augmented baseline.

Meerkat dataset (secondary). Following RoVF's protocol, this tests camouflage, which is a harder case for masked statistics.

MooTrack360 (stress test). We evaluate tracklet re-association across day-to-infrared transitions on its one-hour annotated sequence (MooTrack360). No reported result was found on Lacuna for identity consistency on this dataset, so we report our own baselines only.

Alignment check. Replacing the temporal filter with clip averaging tells us whether texture smearing limits gains on fast-moving animals. If it does, AP3D-style alignment becomes the next step.