Habitat-Keyed Memory: Long-Term Context for Detectors on Moving Cameras

Detectors fail most often on objects that are tiny, blurred, or camouflaged, because the pixels inside the box carry too little information. Two kinds of help are known. The first is the surrounding scene: the same gray blob is more likely a ship on water and a car on asphalt. The second is the past: earlier, clearer sightings of the same kind of object can resolve a bad frame. The best demonstration of the second kind, Context R-CNN, retrieves features from weeks of footage taken by one fixed camera. That retrieval depends on the camera never moving. When the camera is on a vehicle, a drone, or an underwater robot, "same place" stops meaning "same background," and the method has no principled way to decide which past observations are relevant.

We propose to index a detector's long-term memory by what kind of scene it is in rather than by where the camera is. A moving camera revisits the same habitat many times along a transect, such as the same mud, cobble, or road type, even though it never revisits the same pixels. The detector would store past object features together with an estimate of the scene context at the time they were seen. It would then retrieve the stored features whose objects resemble the current one and whose scenes resemble the current scene. The bet is that this makes the "temporal context deficiency" of moving-camera video tractable without optical flow or tracking alignment. Scene context changes slowly and is easy to estimate, so it serves as a stable address for memory even when object appearance is degraded.

Core Claim

On moving-camera video of small, degraded objects, a detector that retrieves from a transect-long memory keyed jointly on object appearance and inferred scene context will beat three alternatives: a single-frame context-aware detector, short-window temporal aggregation, and tracker-based score smoothing. The gains should concentrate on extra-small and low-contrast objects. The key ingredient is the context term in the retrieval key. Removing it, so that memory is keyed only on appearance and recency as in Context R-CNN, should lose most of the gain under ego-motion. Keeping it should not hurt on static cameras.

Where This Sits in Prior Work

Three papers motivate the gap. Context R-CNN adds attention over short-term (burst) and long-term (weeks-long, per-camera) feature memories to Faster R-CNN. It reports mAP gains of 17.9% on Snapshot Serengeti, 19.5% on Caltech Camera Traps, and 4.5% on CityCam. Its long-term memory is meaningful only because the camera is static, and that is exactly the assumption we drop.

Context-Driven Detection of Invertebrate Species in Deep-Sea Video works on moving ROV footage (an ROV is a remotely operated underwater vehicle). It adds a branch that classifies the seafloor substrate into Boulder, Cobble, Mud, or Rock, and uses that as a prior on species. This raises DUSIA mAP from 0.391 to 0.447. Its context is computed independently in each frame, however, and its only temporal component is ByteTrack applied afterwards for counting. We reuse its substrate branch as the source of our memory keys.

Spatio-Temporal Context Modeling for Road Obstacle Detection combines a fixed road-layout prior with Lucas-Kanade optical flow that carries dropped boxes forward. Lucas-Kanade is a classical method that estimates local pixel motion between frames. This raises Faster R-CNN AP50 on the Small Obstacle Detection (SOD) dataset from 48.4% to 52.6%, at about 28 ms extra per frame. Its temporal help is local and short-range, and it cannot recall an object seen a minute earlier.

Several papers supply method or motivation. The Role of Context for Object Detection and Semantic Segmentation in the Wild shows that context matters most for extra-small objects, where it raised accuracy from 2.1% to 7.6%. This is why we report results by size bin. Finding Tiny Faces and Large Selective Kernel Network for Remote Sensing Object Detection obtain context spatially, through very large receptive fields within a single image. They represent the "just look wider" alternative that we must beat.

Two papers constrain the design. Detecting and Recognizing Human-Object Interactions and Something-Else document how background context becomes a shortcut. In Something-Else, I3D drops roughly 15% on compositional splits, where verb-noun pairs at test time never appeared together in training. A context-keyed memory could amplify this shortcut by retrieving "what usually lives on mud." We therefore evaluate separately on out-of-context instances. Who Cares about the Weather? shows that context signatures drift with conditions. That is an argument for context keys that are estimated online rather than fixed.

MovSAM attacks temporal cue deficiency from the opposite side. It uses a multimodal language model to infer motion from a single still image. We do not try to infer dynamics. We substitute retrieved past evidence for the missing frames, which is far cheaper than a pipeline built on several foundation models.

Research Program

The base model is the Faster R-CNN with substrate branch from the deep-sea paper, so that comparisons on DUSIA are exact. Each frame $t$ yields a scene-context embedding $c_t$, taken as the substrate branch's penultimate features. On datasets without substrate labels, the same branch is trained without labels as a global scene embedding. As the camera moves through a transect, the memory stores the region features of confident detections together with the $c_j$ of their frame. For a query region with feature $q$ in frame $t$, attention weights over memory entries $j$ are

$$ w_j \propto \exp\!\left(\frac{q^\top k_j}{\sqrt{d}} + \lambda\, \mathrm{sim}(c_t, c_j)\right), $$

where $k_j$ is the projected stored region feature, $d$ its dimension, $\mathrm{sim}$ cosine similarity, and $\lambda$ a learned scalar controlling how strongly scene context gates retrieval. The retrieved feature is added to the region feature before classification, as in Context R-CNN. Setting $\lambda=0$ recovers an appearance-only long-term memory, which is the central ablation.

To limit drift and error propagation, a known risk for Context R-CNN, memory entries are written only above a confidence threshold. The memory is capped by evicting near-duplicates within each context cluster.

The analysis has three parts. The first breaks gains down by object size, contrast, and how often the context occurs, to test whether retrieval helps where pixels fail. The second inspects which retrieved entries carry high weight, for example whether they come from earlier segments with the same substrate. The third measures behavior on objects that appear in an atypical context, which tests the shortcut risk directly.

Baselines

The naive reference is vanilla Faster R-CNN on DUSIA. The deep-sea paper reports 0.391 mAP for it, and 0.439 with Negative Region Dropping (NRD), a training trick that discards 75% of background proposals.

The strongest existing single-frame competitor is the Context-Driven Detector (CDD) from that paper, at 0.447 mAP on DUSIA. We will reproduce it with the same backbone, NRD setting, data splits, and tuning budget. Beating it means a gain on overall mAP that exceeds the spread across three random seeds, together with a clearly larger gain on the extra-small bin.

The strongest temporal competitor is Context R-CNN run directly on moving footage. We run it twice: once with only its short-term memory over a window of neighboring frames, and once with its full long-term memory ordered by time. Both use the same backbone and memory size as our method. We found no reported result on Lacuna for Context R-CNN on DUSIA or SOD, so we will run it ourselves. Beating it means outperforming both variants, with the margin largest on long transects where useful evidence lies far back in time.

The tracking competitor is CDD followed by ByteTrack, with detection scores averaged along each track. We also run the Lucas-Kanade box propagation from the road-obstacle paper. On SOD that paper reports Faster R-CNN AP50 of 52.6% with its spatio-temporal model, and we reproduce that number as a sanity check before comparing against it.

Two ablations isolate the key ingredient. The $\lambda=0$ variant keys memory on appearance only. The context-smoothing variant uses no memory at all and simply averages $c_t$ over time before it conditions the detector.

Why not just…?

Why not just use Context R-CNN's short-term memory, which already works on moving cameras? This is the most serious threat. Short-term attention over adjacent frames needs no static-camera assumption. However, adjacent frames of a blurred or camouflaged target are usually just as blurred or camouflaged as the current one. Our hypothesis is that the useful evidence is a clear sighting tens of seconds earlier on similar ground. If the short-term-only variant matches our method on DUSIA, the hypothesis is wrong, and we will report that. The size-bin and memory-distance analyses show whether long-range retrieval contributes anything.

Why not just smooth the context prediction over time? Substrate changes slowly, so averaging the context branch's output over time is nearly free and might capture most of the benefit. That would make the memory unnecessary. This is exactly the context-smoothing ablation. If it comes within seed noise of the full model, the honest conclusion is that temporal context integration is sufficient, and the proposal narrows to that cheaper result.

Why not just track and average scores, as the deep-sea counting pipeline does? Tracking helps only once an object has been detected at least once and identity is maintained through the sequence. The deep-sea paper notes identity switches under total occlusion. A tracker cannot recover an object that was never confidently detected, and that is the extra-small regime where Mottaghi et al. found context matters most. The ByteTrack baseline tests this directly, with recall at fixed precision on the smallest bin as the deciding measure.

Why not just enlarge the receptive field? Large kernels and foveal templates give spatial context within one frame. They cannot supply evidence the current frame lacks, and the multi-scale image pyramid in Finding Tiny Faces runs at about 1.4 FPS on 1080p images. We include an LSKNet-style backbone swap at matched compute as a further single-frame baseline.

Proposed Experiments

The primary benchmark is DUSIA, a real moving-camera dataset that comes with substrate labels. We report mAP, AP by object size, AP on an out-of-context subset (species annotated on substrates where they are rare in training), and video counting error. For counting we use the CABOF rule, which counts each track once as it enters the bottom band of the frame. Its reported mean relative error is 0.515, and better detections should not worsen it.

The second benchmark is SOD, a driving dataset without context labels, which tests the learned context keys. We report AP50 against the reported 52.6%.

The control is Caltech Camera Traps, where the camera is static. There our method should match Context R-CNN's reported +19.5% mAP gain within noise, since context and location coincide. A drop there would mean the context term harms the case Context R-CNN was built for.

The decisive checks are the following. On DUSIA and SOD, the full model must beat both Context R-CNN variants, and the $\lambda=0$ ablation must lose most of that margin. The context-smoothing ablation must fall clearly short; otherwise we adopt the simpler result. AP on out-of-context instances must not fall below CDD, which shows that the memory does not merely amplify habitat shortcuts. Latency and memory footprint are reported against the roughly 28 ms Lucas-Kanade overhead.