Underwater cameras and camera traps record far more footage than people can review, and many of the animals in that footage are both rare and camouflaged. Two things are expensive here. Drawing a pixel mask around a fish takes up to two minutes, while clicking a single point on it takes about one second. And the rare species, which matter most for conservation, are exactly the ones whose boundaries a model learns worst. Point-click supervision can already produce good masks for a single conspicuous class. It does this by learning which neighbouring pixels "belong together." That signal comes from appearance, though, so it leaks across the border when a fish has the same colour and texture as the sand. Its appearance cues are also fit mostly to the common species. Our proposal is that a short burst of three consecutive frames contains the missing boundary information for free: pixels on the same animal change together over time, and pixels on the background change differently. We use cheap frame differences to teach the "belong together" model during training only. The final segmenter then runs on single RGB frames, so it still works when the animal stands still and costs nothing extra on a field device.
The two-sentence version is this. Point labels tell the model where an animal is but not where it ends. Co-motion across a few frames tells it where the animal ends regardless of species, so it can supply boundaries for rare classes whose appearance the model has barely seen. The bet is that motion is more useful as a teacher of boundaries than as a test-time input. If that is right, the gains should concentrate on tail species and on low-contrast edges, precisely where both appearance-only weak supervision and class-imbalanced training fail.
Core Claim
Affinity LCFCN trains a segmenter from single clicks. It pairs a class-activation head with an "affinity" head, which predicts how likely each pair of adjacent pixels is to share a label. A random walk spreads the clicks across the affinity graph into pseudo-masks, which then train an ordinary segmenter. It reaches 0.879 mean intersection-over-union (mIoU) on DeepFish. Its affinity head learns only from appearance on static frames, and the file notes that it leaks under extreme camouflage. It is also binary, fish versus background.
Our claim has three parts. First, a motion-derived affinity target, computed from frame differences over a three-frame burst, can supervise the affinity head on pixels where appearance is ambiguous. Second, because this target is class-agnostic and does not depend on how often a species appears, it raises pseudo-mask quality for tail classes more than for head classes. Third, routing motion only into training-time pseudo-label generation beats early input-level fusion of motion channels, the approach of The Motion Picture. The comparison should favour our design most on stationary animals and on footage with noisy background motion.
Long-tail correction of class identity is a separate component. We attach Seesaw Loss to the classification head, so that improvements in boundaries and in class assignment can be measured independently.
Research Program
Motion targets. For each labelled frame we take its two neighbours. Following Balancing Domain Experts for Long-Tailed Camera-Trap Recognition, which found three frames optimal for motion context, the burst is exactly three frames. For handheld footage we first align the neighbours with a global homography. For each pixel $i$ we form a short temporal signature $m_i$ made of the signed differences to the previous and next frame, smoothed over a small window. The affinity target between adjacent pixels $i$ and $j$ is
$$ w_{ij} = \exp\!\left(-\frac{\lVert m_i - m_j \rVert^2}{\sigma^2}\right), \qquad c_{ij} = \min(e_i, e_j)\cdot \rho_{ij}, $$
The symbols are defined as follows. $\sigma$ is a bandwidth set from the median signature distance in each video. $e_i$ is the motion energy at pixel $i$ relative to a per-video noise floor. $\rho_{ij}$ measures whether the local motion field is spatially coherent, which suppresses drifting particles, surface ripple, and swaying weed. The confidence $c_{ij}$ weights a binary cross-entropy between the affinity head's prediction and $w_{ij}$. As a result, motion overrides appearance only where motion is strong and coherent. Elsewhere, the original point-driven affinity loss from Affinity LCFCN governs.
We deliberately use frame differencing rather than learned optical flow. Implicit Motion Handling for Video Camouflaged Object Detection reports drops of more than 10% when camouflage models rely on explicit flow, because flow needs texture to match. The Motion Picture found frame differencing the most consistently effective movement cue on DeepFish. Differencing makes no matching assumption, so it sidesteps the tension between those two results. We test that reading directly with a flow-target variant.
Pseudo-masks and the final model. A random walk over the motion-supervised affinities turns clicks into instance pseudo-masks. A standard instance segmenter is then trained on single RGB frames from those pseudo-masks. Its classification branch uses Seesaw Loss, which scales down the negative gradients that frequent classes impose on rare ones. The final model never sees motion at inference.
Analysis. We stratify every result by three factors: species frequency, using the many/medium/few bins of the DMZ convention (>400, 100–400, <100 instances); local boundary contrast, measured as image gradient magnitude across the ground-truth contour and split into terciles; and whether the instance is moving or stationary within its burst. Each claim above predicts a specific interaction among these factors, so this stratification is what makes the program falsifiable.
Baselines
The naive reference point is a segmenter trained only on single-frame points with plain cross-entropy: the original LCFCN-style point loss, without affinity, motion, or rebalancing. It shows how much any of the machinery buys.
The strongest weakly supervised competitor is Affinity LCFCN with pseudo-mask self-training. It is the only method in the file that turns roughly one-second clicks into masks on low-contrast underwater imagery, and it reports 0.879 mIoU on DeepFish FishSeg. We rerun it from its published configuration on identical clicks, backbone (ResNet38), and schedule, and we confirm we reproduce roughly 0.879 before comparing. Beating it means a statistically reliable gain over seeds in the low-contrast tercile and in few-class mask quality, with no loss on the high-contrast tercile.
The strongest motion-fusion competitor is early input fusion in the style of The Motion Picture: absolute frame-difference channels replace one PCA-reduced colour channel at input. That paper reports a 23% increase in mAP at 50% IoU over raw images on DeepFish detection, and gains for species with fewer than 400 annotations. We apply the same input fusion to the identical point-supervised pipeline, at both pseudo-label and final-model stages, with matched tuning trials. This gives the ablation that isolates our key ingredient: motion in the affinity target only, motion at the input only, and both. Beating it means matching it on moving instances and exceeding it on stationary instances and on the large, noisy datasets where The Motion Picture saw no gain or a slight loss.
The fully supervised upper reference is an ImageNet-initialised segmenter trained on dense masks. A Realistic Fish-Habitat Dataset reports 0.93 mIoU on DeepFish. We aim to close a measurable fraction of the gap between Affinity LCFCN and that number at a fixed annotation-time budget.
On MoCA-Mask the dense-mask reference is SLT-Net. The file gives only its relative margin over RCRNet (9.88% on structure measure); no absolute reported result was found on Lacuna for SLT-Net on MoCA-Mask, so we will rerun it ourselves.
For class identity, the reference is Seesaw Loss in Mask R-CNN. On LVIS v1 it reports rare-class AP of 18.1% versus 0.8% for cross-entropy. That is a different domain, so we use it only as evidence the loss works and compare it within our data against cross-entropy and repeat-factor sampling. A promptable foundation-model baseline is also included; it is described in the next section.
Every method receives the same clicks, the same frames, the same backbone where architectures allow it, and the same number of hyperparameter trials. We report three seeds.
Why not just…?
Why not just feed frame differences into the network, as The Motion Picture does? This is the most serious threat. It is the simplest design, and it demonstrably helps rare species on small datasets. Its own results also show the weakness we target: on Wildcount and Kakadu Fish, motion augmentation gave no significant improvement and sometimes a slight decrease. And an input that depends on motion gives nothing when the animal is still. Our three-way ablation settles the question directly. If input fusion matches our approach on stationary instances and on large datasets, the training-time-only claim is wrong, and we will report that the contribution reduces to motion helping weak supervision in whatever way it enters.
Why not just prompt a promptable segmentation model with the clicks? Turning each click into a mask with the Segment Anything Model is a near-zero-effort pseudo-labeller, so it is a real competitor and we include it as a baseline. The evidence against it is indirect. CaPro finds that off-the-shelf SAM does not segment thin, low-contrast structures without specialised prompt adaptation, and nothing in the file shows it handles camouflaged animals. We will measure its pseudo-mask IoU per contrast tercile. If it wins in the low-contrast tercile, we will instead use its masks as the appearance prior and test whether the motion target still adds on top.
Why not just fix the long tail with rebalancing or copy-paste? Seesaw Loss and context-rich oversampling (CMO, FreeSeg) repair class assignment, but they do not create boundary supervision for a tail species whose edges the model cannot see. Copy-paste also adds hard, artificial edges around pasted animals, the opposite of the soft transitions we need the model to learn. We keep Seesaw Loss as a component and check that it improves tail classification without improving tail boundary quality. If rebalancing alone closes the tail boundary gap, our second claim fails.
Why not just use learned optical flow as the motion teacher? The camera-trap work of Temporal Flow Mask Attention and Balancing Domain Experts shows flow from a pretrained flow network helps rare-species classification. SLT-Net, however, reports that explicit flow degrades camouflage segmentation. We run a flow-target variant with the same confidence gating. Which motion signal survives low contrast is an empirical question, and this comparison answers it.
Proposed Experiments
DeepFish. DeepFish provides point labels on FishLoc, masks on FishSeg, and contrast-heavy habitats. It is the controlled test of boundary quality against Affinity LCFCN. We first verify that the neighbouring video frames of the FishSeg images can be recovered. If they cannot, DeepFish drops to an appearance-only reproduction check.
Tassie BRUV and Kakadu Fish. These supply species-level, long-tailed underwater video. We derive clicks from box centres and annotate dense masks on a held-out evaluation sample stratified by frequency bin and contrast tercile. This sample is the main test of the tail-class claim, and we budget annotation time explicitly in seconds per instance. Wildcount serves as the large-dataset stress test where input-level motion previously failed.
MoCA-Mask. MoCA-Mask provides dense masks for camouflaged video. We simulate clicks and evaluate against SLT-Net. For the long-tail analysis, we attach a per-sequence species label, which is a trivial cost.
Metrics. We report mIoU, a boundary-band F-score measuring agreement within a few pixels of the true contour, mask average precision per frequency bin, and pseudo-mask IoU before final training.
Decisive checks. The first check is a frequency-by-method interaction in pseudo-mask IoU: the motion target should shrink the head-minus-tail gap relative to Affinity LCFCN, not merely shift both up. The second is that the gain concentrates in the lowest contrast tercile. The third is that the RGB-only final model matches input fusion on moving instances and beats it on stationary ones and on Wildcount. The fourth compares frame-difference and flow targets as a function of boundary contrast, to locate where explicit flow stops helping, which is the crossover the SLT-Net and Motion Picture results leave unresolved. If the first and third checks fail together, motion is not acting as a boundary teacher, and the proposal's central claim is refuted.
References
- Affinity LCFCN: Learning to Segment Fish with Weak Supervision
- The Motion Picture: Leveraging Movement to Enhance <scp>AI</scp> Object Detection in Ecology
- Seesaw Loss for Long-Tailed Instance Segmentation
- Balancing Domain Experts for Long-Tailed Camera-Trap Recognition
- Implicit Motion Handling for Video Camouflaged Object Detection
- A Realistic Fish-Habitat Dataset to Evaluate Algorithms for Underwater Visual Analysis
- CaPro: Curvilinear-aware Prompt Learning with Single Unlabeled Image for Cost-effective Curvilinear Structure Segmentation
- The Majority Can Help The Minority: Context-rich Minority Oversampling for Long-tailed Classification
- Learning with Free Object Segments for Long-Tailed Instance Segmentation
- Temporal Flow Mask Attention for Open-Set Long-Tailed Recognition of Wild Animals in Camera-Trap Images