Person re-identification means matching the same pedestrian across different cameras. Models trained on one camera network fail on another, and a large part of that failure is the scenery behind the person. When we adapt a model to a new, unlabeled camera network, the usual recipe is to cluster the target images into "pseudo-identities" and fine-tune on those clusters. That recipe has a quiet failure mode: images from the same camera share pavement, lighting and walls, so the clusters partly group people by camera instead of by identity, and the fine-tuning then reinforces the mistake. Yet the background cannot simply be deleted. The material near a person (a backpack, a companion, the road they walk on) often carries real identity information, and hard foreground masks cut off limbs and belongings.
We propose a cheap, source-free fix that works inside a frozen vision transformer. The transformer cuts the image into patches and turns each into a feature vector, a "token." We estimate a soft foreground weight for each patch from the frozen model's own attention. We then restyle only the background tokens by swapping their per-camera feature statistics for those of another target camera, which keeps their spatial layout. Low-rank adapters (small trainable weight updates added to a frozen network) are trained so that the identity embedding stays unchanged under this swap. The claim is that this background-only, representation-level style transfer removes camera-specific scenery from both the pseudo-labels and the learned features, while keeping contextual cues that pixel-level background suppression throws away. It needs no generator, no source images and no segmentation network.
Core Claim
In source-free adaptation of a frozen re-identification backbone, the dominant error that remains after standard low-rank adaptation is scene-driven pseudo-label clustering. Our hypothesis is that statistics-swapping applied only to background tokens fixes this better than three alternatives: swapping statistics over the whole image, masking out the background, or doing nothing. The specific prediction has two parts. First, our method should lower the fraction of pseudo-label clusters dominated by a single camera. Second, it should raise target mAP over a single-adapter baseline matched in parameters. Applying the same swap to all tokens should help less, because it also perturbs the foreground statistics (clothing colour) that identity depends on.
The proposal sits between two lines of prior work. The first line attacks background shift in pixel space. SBSGAN identified background shift as a main cause of cross-domain degradation. It replaced hard masks with a GAN that softly suppresses the background, at the cost of multiple full GANs and two DenseNet-121 streams. PTGAN kept the pedestrian fixed under a PSPNet mask while CycleGAN restyled the surroundings. Both papers motivate our focus on the background. Both also need raw source images, segmentation, and full fine-tuning, and both treat the background as something to erase or repaint at the pixel level.
The second line is parameter-efficient. SAGE-reID adapts frozen vision transformers to a target network with low-rank adapters trained on clustered pseudo-labels, then merges them with a learned gate. It supplies our training recipe and our strongest baseline. It treats every token identically, however, and has no mechanism against scenery-driven clusters.
Our restyling is a feature-level version of the identity-preserving constraint in SPGAN. SPGAN requires a translated image to stay close to its original in embedding space. We impose the same requirement, but on a background-only perturbation that costs one normalization per token. Two further papers show that background statistics, when separated from foreground ones, can help rather than hurt. GoCNN enforced orthogonal foreground and background filter groups using privileged masks. Beyond Instance Consistency showed that background-only views can train transferable self-supervised features. Both support keeping background tokens rather than discarding them, and we never zero them out.
Research Program
The setting is source-free unsupervised adaptation. We have a vision transformer trained on labeled source data and unlabeled target images with their camera IDs, which re-identification benchmarks routinely provide. Source images are never touched. The backbone stays frozen, and low-rank adapters are inserted into the attention projections as in SAGE-reID, totalling about 2% of the parameters.
Estimating the foreground. For each image, the soft foreground weight $m_i \in [0,1]$ of patch $i$ comes from the frozen class token's attention to that patch in the last layers. We spatially smooth it and rescale it per image. No external parser is used. We will check $m$ against PSPNet masks on a labeled subset, and the check will use the attention maps alone rather than our method's outputs, so the comparison is not circular.
Restyling the background. Let $z_i$ be the token for patch $i$ in an image from camera $c$. Let $\mu_c$ and $\sigma_c$ be the channel-wise mean and standard deviation of background-weighted tokens from camera $c$, computed once from target data at a chosen early block. Let $c'$ be a different target camera drawn at random. The restyled token is
$$\tilde z_i = m_i z_i + (1-m_i)\Big(\sigma_{c'}\tfrac{z_i-\mu_c}{\sigma_c}+\mu_{c'}\Big),$$
which leaves foreground patches unchanged. Background patches receive another camera's colour and texture statistics but keep their spatial layout, so a backpack's shape or a companion's position survives.
Training. Two things use the restyled view. First, clustering into pseudo-labels (DBSCAN, as in SAGE-reID) is run on the average of embeddings from the original and from several restyled views, so that a cluster held together only by shared scenery breaks apart. Second, the adapters are trained with the usual triplet and cross-entropy losses on the pseudo-labels, plus an invariance loss $\lVert f(x) - f(\tilde x)\rVert^2$. Here $f$ is the final identity embedding and $\tilde x$ denotes the image with restyled background tokens. The invariance loss is the feature-level analogue of SPGAN's self-similarity term.
Split adapters. As a secondary ingredient, we test a second adapter that applies only to background-weighted tokens, alongside the main one, at equal total rank. If splitting the adapters does not beat one adapter of the same rank, we drop it. The core claim does not depend on it.
Baselines
The naive reference is direct transfer: the source-trained backbone evaluated on the target with no adaptation. This number shows how much any adaptation buys.
The strongest competitor is SAGE-reID-style single-adapter adaptation. It is the closest system: same frozen backbone, same adapter placement, same pseudo-label loop. Its reported multi-source result is 69.8% Rank-1 and 44.1% mAP on MSMT17, using Duke, Market and CUHK03 as sources (SAGE-reID). We will rerun its recipe ourselves with the identical backbone, adapter rank, clustering hyperparameters, number of epochs and tuning budget, and compare against both our rerun and the reported figure. Beating it means a consistent mAP gain over our rerun across three seeds on at least two of the three transfer settings, with no extra trainable parameters. For the multi-source MSMT17 setting, we would attach our training to each source expert and keep SAGE's merging gate unchanged.
Pixel-level background methods come next. For pairwise transfer, the reported figures are:
| Method | Market→Duke Rank-1 | Market→Duke mAP | Duke→Market Rank-1 | Duke→Market mAP |
|---|---|---|---|---|
| SBSGAN | 53.5% | 30.8% | 58.5% | — |
| PDA-Net | 63.2% | 45.1% | 75.2% | 47.6% |
These use ResNet or DenseNet backbones and full fine-tuning with source access, so they are reference points rather than controlled comparisons. We will report them with that caveat.
Three ablations isolate the key ingredient. The first, whole-image swap, applies the same statistics swap to all tokens and is the test of "background-only." The second, segmentation-mask swap, replaces attention-derived $m$ with PSPNet masks and tests whether self-estimated weights suffice. The third, swap without clustering change, keeps the invariance loss but clusters on original features only, and separates the pseudo-label effect from the training-loss effect.
Why not just…?
Why not just fine-tune standard low-rank adapters on pseudo-labels, as SAGE-reID does? This is the real threat, because a strong frozen backbone may already ignore scenery. Two pieces of evidence point the other way. SBSGAN shows that background shift drives cross-domain errors. Separately, the clustering-and-fine-tuning work behind PUL shows that pseudo-label errors compound over rounds. The deciding measurement is direct: camera purity of clusters under the SAGE-style baseline. If its clusters are already camera-balanced, our mechanism has nothing to fix, and we would report that as a negative result.
Why not just use instance normalization to wash out style, as DualNorm does? DualNorm reaches strong out-of-the-box generalization, for example 60.4% Rank-1 on PRID. It does so by training on five pooled source datasets, and it removes instance statistics everywhere, including the clothing colour that identifies people. We test this directly with a matched adapter variant that adds instance normalization over all tokens at the same block. If it matches us, then the foreground-preserving weight is unnecessary.
Why not just restyle the whole image, MixStyle-fashion, or mix features as IDM++ does? Whole-image perturbation is simpler and might work as well. IDM++ reports that hidden-feature mixing dilutes identity and needs an extra AdaIN module to restore it, which suggests that perturbing the foreground has a cost. The whole-image-swap ablation settles this. If it ties our method, we will say so and recommend the simpler version.
Why not just mask the background with a segmenter? SBSGAN reports that soft masking beat hard masking by 4.7% Rank-1. P²-Net notes that off-the-shelf parsers label bags and umbrellas as background. We include a hard token-masking baseline to confirm this in the frozen-transformer setting.
Proposed Experiments
The transfers are Market-1501→DukeMTMC-reID, DukeMTMC-reID→Market-1501, and multi-source (Duke, Market, CUHK03)→MSMT17. MSMT17's 15 cameras with indoor and outdoor scenes over varying times of day should give the strongest scenery effect, so we expect the largest gains there. Metrics are mAP and Rank-1, plus two diagnostics.
The first diagnostic is cluster camera purity: the share of pseudo-label clusters in which a single camera supplies more than, say, 80% of members. It is tracked over adaptation rounds. The second is a background-swap probe: query images get their background tokens replaced with those of a different target image from another camera, and we measure the resulting drop in retrieval. A model that relies on scenery will drop sharply.
The decisive check has two conditions. Our method must lower camera purity and the swap-probe drop relative to the SAGE-style rerun. It must also beat both the rerun and the whole-image-swap ablation in mAP. If mAP rises but purity does not change, the mechanism is not what we claim. If whole-image swap matches us, the background-specific part is unnecessary.
Two secondary checks follow. One examines Occluded-DukeMTMC queries, where soft weights should avoid discarding occluder-adjacent context. The other compares attention-derived and PSPNet foreground weights. Compute is reported per method as trainable parameters, GPU memory and training time, since avoiding SBSGAN-scale generator pipelines is part of the case.
References
- SBSGAN: Suppression of Inter-Domain Background Shift for Person Re-Identification
- Person Transfer GAN to Bridge Domain Gap for Person Re-Identification
- Low-Rank Expert Merging for Multi-Source Domain Adaptation in Person Re-Identification
- Image-Image Domain Adaptation With Preserved Self-Similarity and Domain-Dissimilarity for Person Re-Identification
- Training Group Orthogonal Neural Networks with Privileged Information
- Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
- Cross-Dataset Person Re-Identification via Unsupervised Pose Disentanglement and Adaptation
- Unsupervised Person Re-identification: Clustering and Fine-tuning
- Frustratingly Easy Person Re-Identification: Generalizing Person Re-ID in Practice
- Bridging the Source-to-target Gap for Cross-domain Person Re-Identification with Intermediate Domains
- Beyond Human Parts: Dual Part-Aligned Representations for Person Re-Identification