Identity-Consistent Memory for Real-Time Person Tracking

Modern single-object trackers keep a short memory of what the target has looked like and match each new frame against it. When the target is a person moving among other people, this memory is the weak point. One occluded or mislabeled frame can write a stranger's appearance into memory, and from then on the tracker follows the wrong person with full confidence. Face-restoration research has a direct answer to a closely related failure. Generators trained only on pixel and perceptual losses produce realistic faces of the wrong person, and the fix is a biometric consistency constraint: a penalty computed in the embedding space of a frozen face-recognition network such as ArcFace, so that outputs must match identity and not only texture. We propose moving that constraint out of image generation and into the tracker itself. The tracker's lightweight memory encoder is trained so its features agree with a frozen biometric network wherever that network is reliable. That agreement then decides which frames may enter memory and which candidate wins during re-detection.

The bet is that identity, not visual similarity, should govern what a person tracker remembers, and that this can be paid for at training time instead of run time. At test time the tracker never calls a face recognizer or a restoration model. It inherits their identity sensitivity through its own features, so it can stay within the roughly 33 ms per-frame budget that real-time tracking needs. That budget rules out the diffusion-based identity restorers in the file.

Core Claim

A tracker whose memory features are distilled toward a frozen biometric embedding, and whose memory writes are gated by agreement with a running identity prototype, will switch to look-alike distractors less often over long person sequences than trackers that gate memory by visual reconstruction error or by a generic triplet verifier. It will do this at near-baseline frame rate and without generating pixels.

Concretely, let $f_\theta(c)$ be the tracker's memory embedding of a target crop $c$. Let $\phi(c)$ be a frozen biometric embedding: ArcFace for face crops (ArcFace), and a body encoder of the kind used in FarSight for full-body crops. Let $q(c)\in[0,1]$ be a reliability weight for the teacher on that crop. We train with

$$\mathcal{L} = \mathcal{L}_{\text{track}} + \lambda\, q(c)\,\big(1-\cos(g(f_\theta(c)),\phi(c))\big),$$

where $\mathcal{L}_{\text{track}}$ is the tracker's usual localization loss, $g$ is a small projection head, and $\lambda$ sets the strength of the constraint. This has the same form as the cosine identity loss in IP-FSRGAN, with one change: the term constrains the tracker's features rather than a generator's pixels.

At run time, a frame is written to memory only if $\cos(g(f_\theta(c_t)), p_t) > \tau$, where $p_t$ is an exponentially averaged prototype of past accepted embeddings and $\tau$ is a threshold. The same score breaks ties among re-detection candidates.

The weight $q$ matters. ArcFace-style spaces degrade under tiny footprints and turbulence, as the file notes for long-standoff imagery. A constraint applied indiscriminately would teach the tracker noise.

Position Relative to Prior Work

Three papers motivate the gap. Generative Target Update for Adaptive Siamese Tracking shows that blocking template updates when something looks anomalous helps a great deal: VOT2017 EAO rises from 0.41 to 0.44 and robustness improves from 0.46 to 0.21. Its anomaly signal, however, is the reconstruction error of a ConvLSTM auto-encoder trained with pixel and adversarial losses. A look-alike person reconstructs well, so the brake does not fire. We keep the brake and replace its signal with identity agreement.

'Skimming-Perusal' Tracking separates localization from a ResNet50 triplet-loss verifier. That verifier is generic, and it adds per-call latency. We fold verification into the tracker's own features and use a biometric teacher in place of generic triplets.

FarSight (Person Recognition at Altitude and Range) couples tracking with recognition-aware restoration, but only in one direction: biometric outputs never feed back into the tracker's association. That system also runs on heavy GPU modules. Our proposal is the reverse arrow, made cheap enough for real time.

Two papers constrain the design. The identity-strong restorers FaceMe and MoDE rely on multi-step diffusion. That is far too slow for per-frame tracking, and MoDE's stochastic completions would flicker frame to frame. This is why we deliberately generate nothing.

The base tracker comes from the SAMURAI line as described in Detector-Augmented SAMURAI: SAM 2 with a motion-aware memory and Kalman filtering. Its memory bank is the object we constrain.

Research Program

We first build the teacher signal. On person-target training sequences we compute $\phi$ on face crops when a face is detectable and on body crops otherwise. We define $q$ from two quantities: crop resolution, and the teacher's own stability under small augmentations, measured as the cosine between embeddings of jittered copies. This makes $q$ low exactly where the biometric space is known to break, without any learned quality model.

Second, we fine-tune only the memory encoder and projection head of the base tracker with the loss above, and keep the mask decoder frozen so localization quality is not traded away.

Third, we add the write gate and the candidate tie-breaker, and calibrate $\tau$ on held-out sequences.

The analysis question running through all three stages is where the gain comes from. Possible sources are fewer corrupted memory writes, better re-acquisition, or the features themselves becoming more discriminative. We log every memory write and label it post hoc as correct-identity or wrong-identity using ground-truth boxes, so contamination rate is measured directly rather than inferred from success curves.

Baselines

The naive reference is the unmodified SAMURAI tracker, which writes every frame to memory with its default policy. We run it with its released weights, at the same input resolution and on the same hardware as all other methods. No reported result was found on Lacuna for SAMURAI on LaSOT or on person-only subsets. We will therefore report our own reproduction and treat it as the floor.

The strongest existing method for long-term robustness with a verifier is SPLT. It reports VOT2018LT F-score 0.616 and OxUvA MaxGM 0.622 at 25.7 FPS (SPLT). We will run it on the identical benchmarks and person subsets, and report FPS on our hardware. Beating it means higher F-score and MaxGM on person-target sequences at equal or higher frame rate.

The closest competitor on the specific mechanism, gating memory updates, is the reconstruction-error brake of Adaptive SiamRPN++ (Generative Target Update), reported at VOT2017 EAO 0.44. To isolate the gating signal from the backbone, we reimplement its brake on top of our SAMURAI base. The ConvLSTM auto-encoder is trained on the same person sequences with the same budget. The only difference from our method is then the signal that decides a write. Beating it means a lower wrong-identity write rate and fewer identity switches at matched success.

We also include detector re-prompting following Detector-Augmented SAMURAI. That work reports success gains up to +0.393 and false-negative reductions up to 0.475 over plain SAMURAI on drone sequences. We run it with a person detector of matched latency.

Three ablations isolate the key ingredient. The first is the teacher-at-runtime gate: frozen ArcFace or body encoder called on raw crops every frame, with no distillation. The second is distillation without the $q$ weighting. The third is distillation with the tracker's own generic features substituted for the biometric teacher. All variants share data, training steps and the $\tau$ calibration protocol.

Why not just…?

Why not just call ArcFace on each raw crop and gate memory with it? This is the most serious alternative, and it is ablation one. If the frozen teacher can be called every frame within budget and matches our accuracy, then the contribution reduces to engineering, and we will say so. We expect it to fall short in two places. The first is latency: SPLT's experience with a ResNet50 verifier shows that per-frame heavy verification is what pushes systems toward sparse, heuristic invocation. The second is reliability: raw degraded crops weaken biometric matching, with LFW verification at 92.8% on raw low-resolution faces versus 97.6% after identity-preserving restoration in IP-FSRGAN, and ArcFace at 64.6% on occluded LFW in MoDE. A raw-crop gate will therefore veto good frames or admit bad ones exactly when the target is small or occluded. Accuracy stratified by crop size will show whether this happens.

Why not restore crops first, FarSight-style, then verify? Restoration with an identity loss does recover identity, as both IP-FSRGAN and FarSight show. But the strongest restorers run at hundreds of milliseconds to seconds per crop, and a restorer can hallucinate a plausible face that then contaminates memory. We include an IP-FSRGAN-style restore-then-gate variant on a subset of sequences to measure its latency and write accuracy rather than assume either.

Why not just re-prompt with a person detector? Detectors return "a person," not "this person." On sequences where the target crosses paths with similar people, re-prompting picks the nearest box. We expect it to win on exit and re-entry sequences without distractors and to lose on crossing events, and we report the two strata separately.

Why not keep the reconstruction-error brake? A look-alike reconstructs as well as the target. The matched reimplementation described under Baselines tests this directly on distractor-crossing events.

Proposed Experiments

We evaluate on LaSOT (LaSOT) and OxUvA (Long-term Tracking in the Wild), restricted to sequences whose target is a person. Within these, we annotate the frames where another person overlaps or passes within a fixed distance of the target. We add VOT2018LT for comparability with SPLT. Aerial transfer is tested on VisDrone pedestrian tracks, where the reference point is MM-Tracker's IDF1 of 58.3% at 31.1 FPS (MM-Tracker). That tracker is multi-object, so this comparison is indicative only. Metrics are success and precision, MaxGM, F-score, identity switches on distractor-crossing events, wrong-identity memory write rate, and FPS on a single GPU and on an embedded-class device.

Three checks decide the outcome. The first is whether wrong-identity writes drop sharply relative to both the plain tracker and the reconstruction-error brake on crossing events. If they do not, the mechanism is not doing what we claim. The second is whether the distilled gate matches the teacher-at-runtime gate on accuracy while staying above 25 FPS. If the teacher is equally accurate and fast enough, distillation is unnecessary. The third is whether removing $q$ hurts most on the smallest crops, which would confirm that the biometric constraint must be applied selectively. If gains appear only on large, frontal-face sequences, we would conclude that the benefit is confined to near-range tracking and report it as such.