Person re-identification means matching a pedestrian seen by one camera to the same person seen by another. Systems that do this well usually split the person into parts (head, torso, legs, bag) and compare part to part, because one global vector washes out small cues like a logo or a shoe. The oldest way to get parts is to cut the image into fixed horizontal stripes. Stripes break when people bend, when the detector's box is off-center, or when something blocks the view. Image-only models such as AAformer fix this by letting each image patch be assigned to a learned part prototype through optimal transport, a balancing procedure that spreads patches evenly across parts. Video, though, is where re-identification is actually deployed, and the strongest video models either keep rigid stripes or skip parts altogether. Doing part discovery independently in each frame lets a part's meaning drift from frame to frame. Filtering by motion, the other common trick, cannot tell a swinging arm from a passer-by walking through the box.
We propose to solve the part assignment once per tracklet (the short sequence of cropped frames a tracker produces for one person) rather than once per frame. All patches from all frames are distributed across a single shared set of part prototypes plus a "discard" slot, and the parts are not forced to receive equal mass in every frame. A patch that moves but keeps matching the same part prototype is the person's own limb in motion. A patch that matches no part anywhere in the tracklet is an occluder or background, and it goes to the discard slot. Both kinds of patch move, so the decision rests on what a patch looks like relative to the whole tracklet, not on whether it moves. The patch features feeding this assignment come from one network with convolutions in its tokenizer, so the model gets the sharp local texture that convolutions provide without running two full backbones side by side.
Core Claim
Assigning patches to parts jointly over a whole tracklet, using unbalanced transport with a discard slot, should keep part identity stable across frames and separate the target's own movement from external occlusion. We expect it to beat both rigid stripes and frame-independent part clustering on occlusion-heavy video benchmarks. If per-frame clustering followed by temporal pooling performs just as well, the claim is wrong, and the experiments are built to show this directly.
Concretely, let a tracklet have $T$ frames of $N$ patches, with patch features $x_{t,n}$ and $K$ learnable part prototypes $p_k$ plus one discard prototype $p_0$. We solve for an assignment $Q \in \mathbb{R}^{(K+1)\times TN}$:
$$ \max_{Q \ge 0}\; \sum_{k,t,n} Q_{k,(t,n)}\, \langle p_k, x_{t,n}\rangle + \epsilon H(Q) \quad \text{s.t.}\; \sum_k Q_{k,(t,n)} = \tfrac{1}{TN},\;\; \sum_{t,n} Q_{k,(t,n)} \ge \tfrac{\rho}{K}\ \text{for } k\ge 1. $$
Here $H$ is the entropy that makes the problem solvable by Sinkhorn iterations, $\epsilon$ sets how soft the assignment is, and $\rho<1$ is the minimum share of the tracklet each real part must claim. The balance constraint is applied over the whole tracklet rather than inside each frame. A part hidden in three frames can therefore receive little mass there and make it up in the frames where it is visible. AAformer's per-image equal-partition constraint cannot do this, because it forces every part to be found in every image.
Research Program
The backbone is a single-stream vision transformer whose linear patch projection and attention projections are replaced by convolutional ones. This follows the convolutional token embedding and projection design of ViTransPAD, which used it for face spoof detection with fixed multi-scale grids and no part reasoning. We use that design only as the source of local inductive bias. It is the cheap alternative to the parallel ResNet-50 and ViT-Base pair in DCCT, which reports that coupling the two streams adds about 2.6% mAP on MARS but needs distillation to be affordable at inference.
In the last few blocks, part tokens attend only to the patches assigned to them, as in AAformer. The assignment comes from the tracklet-level transport problem above instead of a per-image one. Each part's final descriptor is pooled over the whole tracklet, weighted by its transport mass. Low-mass frames therefore count less for that part only, not for the whole frame. This differs from the frame-level weighting in DCCT and in the visibility regressor of TCViT.
Training uses identity and triplet losses on each part descriptor and on a global token, as in PCB. There is no pose or parsing supervision.
The analysis has two parts. First, we measure part-correspondence drift: how often the same body region changes part label between adjacent frames. We compare tracklet-level transport, per-frame transport, and rigid stripes. Second, we run an articulation-versus-occlusion diagnostic. We paste moving pedestrian crops onto clean MARS tracklets with known masks, which gives ground-truth occluder pixels. For evaluation only, we run a pose estimator on a held-out subset to locate the target's limbs. We then report the fraction of occluder patches sent to the discard slot and the fraction of the target's limb patches wrongly discarded. The same two fractions are computed for TCViT's correlation-based suppression, which is the method the file's Gap 3 suspects of discarding swinging limbs.
Baselines
The naive reference point applies rigid six-stripe pooling, as in PCB, to each frame of the same convolutional-token ViT and averages across frames. It shows what any part machinery must improve on. In the image setting, PCB with refined part pooling reaches 93.8% Rank-1 and 81.6% mAP on Market-1501. We will reproduce it on video ourselves.
The strongest existing methods are TCViT and DCCT. On MARS, TCViT reports 87.6% mAP, and on LS-VID it reports 83.1% mAP and 90.1% Rank-1. TCViT is the most direct competitor on the occlusion question, because it is the published motion-based occluder filter. DCCT reports 87.5% mAP and 92.3% Rank-1 on MARS and 97.6% mAP on DukeMTMC-VideoReID. It is the strongest hybrid convolution-transformer model. ST-MGA (87.2% mAP on MARS, 79.3% mAP on LS-VID) and STT (86.3% mAP on MARS) represent learned-attention parts and rigidly constrained parts in video.
We will report each method's published numbers. We will also rerun TCViT and DCCT, plus a per-frame AAformer variant, in a shared codebase with the same ImageNet initialization, 256×128 crops, 8-frame clips, epoch budget, and hyperparameter search budget. No method gets synthetic pre-training unless all of them do. AAformer has no video result to compare against: no reported result found on Lacuna for AAformer on MARS or LS-VID.
The key ablation swaps only the transport scope. The main variant solves one problem per tracklet with the discard slot. The control solves one balanced problem per frame, which is exactly AAformer's constraint, followed by temporal averaging. Everything else is held fixed.
Beating the field means two things. On LS-VID, mAP must exceed TCViT's reported 83.1% and also exceed our reproduction of TCViT. On MARS, the model must match or beat 87.6% mAP while using fewer FLOPs than DCCT's coupled teacher. On DukeMTMC-VideoReID, scores near 97–98% mAP leave little room to improve, so we treat it as a sanity check and not as evidence.
Why not just…?
Why not just run AAformer on each frame and average? This is the real threat. Temporal averaging might wash out per-frame part drift well enough that tracklet-level transport adds nothing. The transport-scope ablation settles it directly. If per-frame transport matches the tracklet version on LS-VID mAP and on the drift and occluder-discard diagnostics, we will report that the gain comes from parts themselves and not from temporal coupling. We would then drop the coupling claim.
Why not just keep rigid stripes, since tracklets average out jitter? AAformer gains 4.1% Rank-1 over its baseline on CUHK03's automatically detected (misaligned) crops. STT needed a spatial constraint and synthetic pre-training even to train stably. MARS and LS-VID tracklets come from automatic detectors and trackers, so box drift persists across frames rather than averaging out. The stripe baseline above measures how much of this matters in video.
Why not just filter occluders by motion, as TCViT does? It works, and its LS-VID numbers are the bar we must clear. Its premise is that the target is static within the crop and occluders move, which predicts that the target's moving limbs will be suppressed. The synthetic-occluder diagnostic tests this prediction directly. If TCViT turns out not to discard limbs, our main argument against motion filtering weakens, and we will say so.
Why not just use a pose estimator or human parser? The file reports that parsers miss non-body cues such as bags (AAformer) and fail in exactly the crowded, occluded scenes where re-identification is hardest (SCFP). Running one on every frame of every tracklet is also costly. We use pose only offline, for evaluation.
Proposed Experiments
The main evaluation runs on MARS and LS-VID, with DukeMTMC-VideoReID, iLIDS-VID and PRID-2011 as secondary benchmarks. We report mAP and Rank-1, along with FLOPs and latency per tracklet.
The robustness evaluation covers clean MARS tracklets with synthetic occluders pasted in at controlled coverage levels. We report retrieval mAP as a function of occlusion level, plus the occluder-discard and limb-discard rates. The static-image check runs the same model with $T=1$ on Occluded-Duke against AAformer's 67.1% and SCFP's 70.9% Rank-1. Its purpose is to confirm that dropping the per-frame equal-mass requirement does not hurt single images.
Three outcomes would falsify the proposal, and any one of them would be decisive. The first is that per-frame transport ties tracklet-level transport. The second is that the discard slot absorbs limbs as often as TCViT suppresses them. The third is that the single-stream convolutional tokenizer falls behind DCCT by more than DCCT's own 2.6% coupling gain. In that last case, the efficiency argument does not hold.
References
- Vitranspad: Video Transformer Using Convolution And Self-Attention For Face Presentation Attack Detection
- Deeply Coupled Convolution-Transformer With Spatial-Temporal Complementary Learning for Video-Based Person Re-Identification
- AAformer: Auto-Aligned Transformer for Person Re-Identification
- Temporal Correlation Vision Transformer for Video Person Re-Identification
- Beyond Part Models: Person Retrieval with Refined Part Pooling (and A Strong Convolutional Baseline)
- Multi-Granularity Aggregation with Spatiotemporal Consistency for Video-Based Person Re-Identification
- Spatiotemporal Transformer for Video-based Person Re-identification
- An end-to-end occluded person re-identification network with smoothing corrupted feature prediction