Tracking Look-Alike Targets with Per-Video Centerless Projections

When a tracker follows many people who look almost the same, such as dancers in matching costumes, athletes in team kits, or workers in uniforms, the appearance features it relies on stop telling people apart. The DanceTrack benchmark showed this clearly: re-identification embeddings (feature vectors trained so that crops of the same person land close together) become entangled when everyone wears similar clothing. In that regime, using them can do worse than matching on box overlap alone (DanceTrack). The strongest current answer, Deep OC-SORT, does not repair the embedding. It detects when appearance is ambiguous and hands the decision to motion through hand-set gating rules. Motion in turn fails under non-linear movement and long occlusions (OC-SORT).

We propose to repair the embedding for each video while the tracker runs. In any given video, the generic re-identification space is dominated by what all the people share, so the small differences that separate them are drowned out. A cheap linear projection, fit on that video alone, can amplify those differences. Two people detected in the same frame are guaranteed to be different identities, which gives free and nearly error-free supervision. The projection is fit with a centerless objective, meaning one written only over differences between pairs of detections and never over an average "track prototype." The tracker then alternates between two steps: fixing the current associations and refitting the projection, then fixing the projection and re-associating.

Core Claim

We claim that most of the identity information lost in uniform-appearance scenes is still present in the generic embedding, only at low variance. We further claim that a per-video discriminant projection can recover it online with closed-form updates. The projection must have three properties. It must be learned from within-frame "cannot-link" pairs and high-confidence "must-link" pairs. It must use a robust, pairwise objective rather than centroids. It must be refit in alternation with association. We expect it to raise association accuracy on DanceTrack above Deep OC-SORT without slowing the tracker materially.

We want to be precise about what "centerless" buys here, because the identity in the research file cuts against a naive version of the claim. With a squared Euclidean norm and all pairs included, pairwise scatter equals scatter about the mean, so a centerless objective would be algebraically identical to centered LDA. The difference we rely on comes from two departures from that special case. First, we restrict and weight the pairs: cannot-link pairs exist only between people who appear together in a frame, and no single centroid can express that graph. Second, we use a robust norm on those pairs, which corresponds to no centroid at all. Blurred or half-occluded crops then cannot drag a track prototype away. Deep OC-SORT identifies exactly this kind of corruption as a failure of averaged appearance profiles.

Research Program

The method draws on a line of centerless dimensionality-reduction work, and each paper plays a specific role. Avoiding Optimal Mean Robust PCA/2DPCA and its $\ell_{2,p}$ generalization (Robust Formulation for PCA) replace variance about the mean with robust sums over projected pairwise differences. These two papers supply our robust norm. Self-Weighted Robust LDA replaces the global mean in between-class scatter with pairwise class distances, so that far-away "edge classes" stop dominating the learned subspace. In our setting, the edge class is the one dancer in a different costume, who would otherwise dominate the projection while the look-alikes stay merged. This motivates the concave weighting on between-identity pairs described below. Enhanced Unsupervised Discriminant Dimensionality Reduction alternates between a linear projection and centerless cluster assignments, and it names temporal, dynamic data as future work. We reuse its alternation pattern, but our assignments come from a tracker with motion constraints and cannot-links rather than from $K$-means. CMKOA, Unified K-means, and DPAC show that centerless pairwise objectives resist collapse and outliers on static data. DPAC also reports degradation when initial features are highly similar. That warning constrains us: a purely unsupervised centerless clustering of entangled embeddings is expected to fail, which is why we inject the within-frame cannot-links.

The objective works as follows. Let $x_i$ be the generic re-identification embedding of detection $i$, first reduced once per video to a moderate dimension $D$. Let $W \in \mathbb{R}^{D \times r}$ be the projection, with orthonormal columns ($W^\top W = I$). Let $\mathcal{C}$ be the set of detection pairs that co-occur in a frame within a sliding window. Let $\mathcal{M}$ be the set of pairs linked in the same track by high-confidence associations. We solve

$$\max_{W^\top W = I} \; \sum_{(i,j)\in\mathcal{C}} \|W^\top(x_i - x_j)\|_2^{q} \;-\; \lambda \sum_{(i,j)\in\mathcal{M}} \|W^\top(x_i - x_j)\|_2^{p},$$

where $q<2$ gives extra weight to the hardest, closest cannot-link pairs (the look-alikes), in the spirit of the self-weighted LDA idea. The exponent $p<2$ makes within-track agreement robust to bad crops, and $\lambda$ balances the two terms. We handle this objective by iterative reweighting, following the robust-PCA papers. Each pair receives a weight proportional to its current projected distance raised to the power $p-2$ (or $q-2$). The subproblem then becomes a top-$r$ eigen-decomposition of a weighted scatter-difference matrix of size $D\times D$. That step is closed form and costs little when $D$ is on the order of a hundred.

The tracker alternates two blocks. With $W$ fixed, association runs as in OC-SORT, keeping its observation-centric motion terms and its Hungarian matching. The appearance term is computed in the projected space. With the associations fixed, $\mathcal{M}$ is rebuilt from matches above a confidence margin, and $W$ is refit every few frames. The wireless alternating-optimization literature supplies a design constraint here rather than a method. Zhu et al. report 58 seconds for an SDR-based alternating solver against 2.3 seconds for a first-order one at nearly the same quality. Both are far beyond a video frame budget, so every block in our loop must be closed form or a handful of reweighting steps, never a relaxation solved to optimality.

This design is what makes the method scale with the number of targets. The within-frame cannot-link pairs grow quadratically in the number of people visible at once. That growth is manageable on DanceTrack but not in MOT20 crowds, so we subsample pairs by projected distance, keeping the nearest confusable ones. We will report how runtime and accuracy vary with the number of concurrent targets.

The main risk is self-reinforcement: a wrong association becomes a wrong must-link, which pulls two identities together, which causes the next wrong association. Cannot-links do not suffer this, because they depend only on the detector not producing duplicate boxes. We therefore test a cannot-link-only variant. We also test an oracle variant, in which $W$ is fit from ground-truth identities on the same video. The oracle gives an upper bound on what any linear per-video projection can do.

Baselines

The naive reference points are SORT and OC-SORT, which use motion and overlap only. OC-SORT reports 54.6 HOTA on DanceTrack against SORT's 47.9, and 62.1 HOTA on MOT20. HOTA (Higher Order Tracking Accuracy) combines detection accuracy with association accuracy. These systems tell us what appearance must add at all. We also run OC-SORT plus the raw, unprojected re-identification distance with a fixed weight. This is the "do nothing to the embedding" point, and DanceTrack predicts it will be weak.

The strongest existing method is Deep OC-SORT. It adds camera-motion compensation, confidence-gated appearance updates, and adaptive weighting that shifts trust to motion when appearance candidates are ambiguous. It reports 61.3 HOTA on DanceTrack, 64.9 on MOT17, and 63.9 on MOT20. It is the right competitor because it attacks the same entanglement problem, only with gating instead of representation repair. For a fair comparison, we build our method on Deep OC-SORT's own code path, with the same detections, the same re-identification backbone, and the same motion model. We replace only the appearance distance, and we give both methods the same validation-split tuning budget over their thresholds. Beating it means exceeding 61.3 HOTA on the DanceTrack test split, with the gain showing up in association accuracy (AssA) and IDF1 while detection accuracy (DetA) stays essentially fixed. Because the detections are shared, DetA should not move, so any HOTA gain that is not an AssA gain would be suspect. We found no reported AssA or IDF1 values on Lacuna for Deep OC-SORT on DanceTrack, so we will reproduce them ourselves under the shared setup.

Several ablations isolate the proposal's ingredients. The centered ablation fits standard LDA using exponentially averaged track prototypes, with a squared norm, on the same must-link and cannot-link data. It tests whether the centerless, robust formulation matters at all. The one-shot ablation fits $W$ once from the first high-confidence tracklets and never refits, which tests the alternation. The cannot-link-only and oracle-$W$ variants described above bound the method from below and above.

We considered ADA-Track, which alternates detection and track queries inside a transformer and reports 0.500 AMOTA on nuScenes. It is not a direct baseline, because it targets multi-camera 3D driving scenes at 300–450 ms per frame.

Why not just…?

Why not just fine-tune the re-identification network on each video with the same cannot-link pairs? This is the most serious threat, and we treat it as a co-equal baseline rather than dismissing it. Test-time contrastive fine-tuning of the backbone uses the same free supervision with far more capacity. Our bet is that tens of frames of look-alike crops are too few to update millions of weights without overfitting or drifting. We also bet that a $D\times D$ eigen-solve every few frames is orders of magnitude cheaper than backpropagation. We will run fine-tuning of the last backbone block under a matched wall-clock budget. If it matches our AssA at comparable latency, the contribution shrinks to a cheap approximation, and we will report it that way.

Why not just whiten or PCA the embeddings per video? Subtracting the scene mean and whitening would remove the shared "costume" direction without any labels. We expect this to help somewhat, and it is included as a baseline. It does not know which directions separate co-present people, however, and whitening can amplify noise directions as easily as identity directions. If whitening recovers most of the gain, the discriminant and centerless parts are unnecessary, and the experiment will say so.

Why not just trust Deep OC-SORT's adaptive weighting and fall back on motion? Gating decides whether to use appearance. It cannot make appearance informative again, and the fallback inherits motion's documented failure on non-linear, long-occlusion segments (OC-SORT). We will stratify DanceTrack identity switches by occlusion length. If our gains are not concentrated in exactly the segments where Deep OC-SORT hands control to motion, the mechanism we claim is wrong.

Why not just fit the projection once instead of alternating? A one-shot fit may already capture most of the separable directions, which would make alternation an unneeded complication. The one-shot ablation measures this directly, broken down by video length. Alternation should matter most in long videos, where occlusions create new confusions late.

Decisive Experiments

The first check, and the kill switch, is the oracle test on DanceTrack validation. We fit $W$ from ground-truth identities in each video and measure two things: the ratio of intra-identity to inter-identity projected distance among co-present people, and AssA. If even the oracle projection cannot improve AssA over Deep OC-SORT, the identity signal is absent from the linear span of the embedding, and the DanceTrack finding stands as a hard limit. In that case we would stop.

If the oracle passes, the main experiment runs the full method on the DanceTrack test split against Deep OC-SORT's 61.3 HOTA, OC-SORT's 54.6, and SORT's 47.9, reporting HOTA, AssA, DetA, IDF1, and per-frame latency. The size of the gap between the oracle and the cannot-link-only variant tells us how much the method depends on its own associations, and hence how exposed it is to self-reinforcing errors.

The two remaining checks test generality and scale. On MOT17 (Deep OC-SORT: 64.9 HOTA) and MOT20 (63.9), where appearance is more varied, the method must not regress. On MOT20 we also plot runtime and AssA against the number of concurrent targets, with and without cannot-link subsampling, to confirm that the quadratic pair cost is controlled. Finally, the centered ablation separates the contribution of the robust pairwise objective from that of the supervision signal. That separation is what justifies calling this a centerless method rather than LDA with free labels.