Matching the same person between a drone camera and a street-level camera is hard because a person seen from 30 meters overhead looks less like themselves from the front than two different people look from the same angle. The current best systems for this aerial-ground re-identification task split into two camps. One camp strips viewpoint out of the representation wholesale, by subtracting a learned "view" vector or by penalizing any correlation with it. The other camp keeps viewpoint and pays for it with a diffusion model that imagines the missing view. The first camp risks throwing away real identity cues that happen to be visible only from one angle, such as a hat, a hair parting, or a backpack strap seen from above. The second camp is expensive and can invent details.
This note proposes a third route that is cheaper than either and can be inspected. Many strong aerial-ground models are built on CLIP, an image encoder trained to share an embedding space with a text encoder. Prior interpretability work showed that CLIP's final image vector is a sum of contributions from individual attention heads, and that each head's contribution can be written in terms of text directions. We propose to use this decomposition to find which heads encode camera geometry ("photo taken from above," "top of a head," "tilted perspective"). We then remove viewpoint only in those heads, at test time and without retraining. Heads whose text directions describe clothing, color, or carried objects are left alone, even when their outputs correlate with viewpoint. The two-sentence version is this: aerial-ground re-identification models discard viewpoint too coarsely, and CLIP's language-aligned head decomposition gives a way to tell nuisance geometry apart from view-correlated identity evidence. We test whether cutting only the geometry heads beats both blanket invariance and generative view imagination.
Core Claim
Our claim is that in CLIP-based aerial-ground re-identification models with a frozen backbone, viewpoint information is concentrated in a small set of late attention heads. Those heads can be picked out by projecting their outputs onto text directions. Suppressing them, or projecting the view-describing text directions out of them, will improve cross-view Rank-1 and mean average precision (mAP). It should do so more than an equal-budget suppression chosen without language, and it should keep identity cues that blanket invariance destroys.
The decomposition we reuse can be written as
$$z = \sum_{l=L-3}^{L} \sum_{h=1}^{H} c_{l,h} + r,$$
where $z$ is the image embedding, $c_{l,h}$ is the direct contribution of head $h$ in layer $l$ projected into CLIP's joint space, $L$ is the last layer, and $r$ collects the earlier layers and MLP terms, which the original analysis found to matter little. The edit we propose is
$$c'_{l,h} = c_{l,h} - P_{V} \, c_{l,h} \quad \text{for } (l,h) \in S,$$
where $P_V$ is the orthogonal projector onto a small set of text-derived viewpoint directions, and $S$ is the set of heads whose text spans are dominated by viewpoint phrases. The hard version of the edit replaces $c_{l,h}$ with its dataset mean for heads in $S$.
Where This Sits in Prior Work
The method comes from Interpreting CLIP's Image Representation via Text-Based Decomposition. That paper decomposes CLIP-ViT head outputs into text directions (TEXTSPAN) and finds that the representation forms mostly in the last four attention layers. It also shows that zeroing "geolocation" heads raises Waterbirds worst-group accuracy from 48% to 73% without retraining. It was never applied to person re-identification or to camera geometry. Our proposal treats aerial viewpoint as the analogue of Waterbirds' spurious background. There is one important difference. In re-identification the nuisance variable is partly informative, so the question is not only whether it can be removed but whether it can be removed selectively.
The motivation comes from the split described in the opening. View-decoupled Transformer for Person Re-identification under Aerial-ground Camera Network subtracts a learned view token from an identity token in every block and adds an orthogonality loss. SeCap keeps that decoupling and adds image-specific visual prompts, and View Confusion Feature Learning does the same job adversarially. All three remove viewpoint from the whole feature, with no way to say what was removed. SD-ReID argues directly that this discards useful overhead cues. It keeps them by generating counterpart-view features with latent diffusion, at the cost of multi-step inference. Our proposal accepts SD-ReID's diagnosis and rejects its remedy: if view-correlated identity cues live in different heads from pure geometry, a subtractive edit restricted to the geometry heads keeps them without generation.
The closest existing system, and our host model, is LATex. LATex freezes a CLIP backbone and adds prompt tokens and attribute classifiers. It aligns images to templates such as "A [UAV] view photo of a [male with an orange T-shirt]." LATex uses text as a training target for the pooled embedding. We use text as a coordinate system for reading and editing the internal head contributions. The two approaches stack, and because LATex keeps the backbone frozen, CLIP's head-to-joint-space projections stay meaningful. AG-ReID.v2 also offers interpretability, through attention maps tied to 15 annotated attributes. Our decomposition needs no attribute labels and is not limited to a fixed attribute taxonomy.
One line of work constrains the proposal. Deep Pre-Alignment for VLMs documents that visual tokens sit far from text geometry after shallow projection, a gap between image and text embeddings that the vision-language literature calls the modality gap. This warns that text directions may explain only a fraction of a head's variance on low-resolution surveillance crops. We therefore treat reconstruction quality as a gating measurement, not an assumption. The CLIP analysis gives a reference point: 60 text directions per head kept 72.77% versus 75.25% ImageNet accuracy. If text spans capture far less than that on re-identification crops, the language labels are unreliable, and we will report this.
Research Program
The first stage is diagnostic. We take LATex's frozen CLIP ViT-L backbone with its learned prompts, and a plain CLIP-ReID-style model as a second host. We compute per-head direct contributions on AG-ReID.v1, AG-ReID.v2, and CARGO. For each of the last four layers' heads we measure three things. The first is how well a linear probe predicts the camera platform (aerial, CCTV, wearable) from that head alone. The second is how much identity information the head carries, measured by leave-one-head-out drop in cross-view mAP. The third is the head's TEXTSPAN description over a vocabulary we write in advance. The vocabulary will contain viewpoint and geometry phrases, clothing and color phrases, carried-object phrases, and hair and headwear phrases, fixed before looking at results.
The central empirical question is whether viewpoint and identity are concentrated in partly different heads, or smeared across all of them. Smearing would sink the proposal.
The second stage is the test-time edit. Heads are ranked by the share of their text-span variance explained by viewpoint phrases. We then apply mean-ablation or the projection edit above to the top $k$ heads, sweeping $k$ from 1 to 16. We choose $k$ on a held-out split of the training identities, never on the query and gallery sets.
The third stage is a cheap training variant. It adds a view-prediction penalty only on the selected heads during LATex's prompt tuning, to test whether localized invariance beats VDT's global orthogonality under matched training.
The fourth stage checks what is preserved. We build a subset of cross-view query pairs in which the identity-distinguishing cue is visible mainly from above. Using AG-ReID's attribute annotations, these are identities that differ mainly in headwear or hair. Selective editing should hold accuracy on this subset, while blanket removal should lose it.
Baselines
The naive reference point is the unedited host model: LATex's frozen CLIP backbone with prompts, run with its released training recipe on each benchmark and evaluated under each benchmark's standard cross-view protocol. Every edit is applied to this same checkpoint, so any gain is attributable to the edit alone.
The strongest existing competitor is LATex itself, because it is the best reported system in the file on the benchmarks we use. It reports 88.88% Rank-1 and 79.19% mAP on AG-ReID.v1 Ground-to-Aerial, and 90.18% Rank-1 on CARGO with pseudo-attributes. Protocols across papers are not uniform: GSAlign reports 64.89% Rank-1 on CARGO's aerial-to-ground protocol, and SD-ReID reports 70.48%. We will therefore reproduce LATex under each benchmark's stated protocol and compare against our own reproduction, citing the published numbers only as sanity checks. Beating it means a consistent gain in both Rank-1 and mAP across at least three benchmark-protocol pairs, larger than the seed-to-seed spread over three training seeds.
For view-invariance training we compare against VDT's subtractive decoupling with its orthogonality loss, retrained on the same CLIP backbone with the same data, epochs, and a matched hyperparameter search. On LAGPeR, the harder real benchmark, we compare against SeCap's reported 41.79% Rank-1 and 30.37% mAP in the aerial-to-ground setting. We compare against SD-ReID where its numbers exist, 85.57% Rank-1 on AG-ReID.v1, and also report its inference cost relative to ours, since cost is part of the claim.
Two ablations isolate the key ingredient. The first ablates $k$ randomly chosen heads to rule out generic regularization. The second selects heads by the view-probe accuracy described above, with no text involved. The comparison between text-selected and probe-selected heads is the decisive one. It asks whether projecting into language space adds anything beyond a purely visual signal.
Why not just…?
Why not just project out the directions a linear probe finds predictive of camera platform? This is the most serious threat. Removing linearly view-predictive directions from the pooled embedding is simpler and needs no text. However, any direction that predicts viewpoint gets removed, including a headwear direction that is simply more visible from above, which is exactly SD-ReID's objection to invariance. We run this nullspace baseline at matched rank. We expect it to match our edit on overall accuracy but to lose on the overhead-cue subset. If it also holds up on that subset, the language projection contributes interpretability but not accuracy, and we will say so.
Why not just fine-tune the whole backbone end to end? Full fine-tuning can absorb viewpoint shift. But LATex already reaches its results with a frozen backbone and about 36M trainable parameters against 122M, so the frozen regime is competitive rather than a handicap. Full fine-tuning also pulls head outputs away from CLIP's text space, and Deep Pre-Alignment's results on shallow projection suggest that shallow alignment leaves visual tokens far from text geometry, which would make any text-based reading of the heads meaningless. We include a fully fine-tuned CLIP as a reference and measure TEXTSPAN reconstruction on it to show whether the decomposition survives.
Why not just use VDT-style orthogonality or SD-ReID-style generation, which already work? Both work, and we compare against both. Orthogonality removes view information from the entire vector with no record of what was lost. Generation keeps the information but adds diffusion steps at inference. Our edit costs a single subtraction per selected head at inference and names what it removes. The comparison on the overhead-cue subset, together with wall-clock inference time, settles whether that trade is worth making.
Decisive Checks
The proposal fails early if the diagnostic stage shows no concentration. That would appear as view-probe accuracy spread evenly across heads, or as text spans explaining too little head variance to trust the labels. Both are reported as findings. It fails at the main test if text-selected ablation does not beat random ablation at equal $k$ on AG-ReID.v2 and CARGO.
It succeeds most clearly if four things hold at once. Text-selected editing should improve cross-view mAP over the unedited LATex checkpoint. It should match or beat retrained VDT-style orthogonality. It should keep accuracy on the overhead-cue subset where linear-nullspace removal loses it. And it should add negligible inference cost compared with SD-ReID.
A final transfer check applies the head set selected on AG-ReID.v2 unchanged to LAGPeR and G2APS-ReID. If viewpoint heads are a stable property of the CLIP backbone rather than of one dataset, the gains should carry over without reselection.
References
- Interpreting CLIP's Image Representation via Text-Based Decomposition
- View-decoupled Transformer for Person Re-identification under Aerial-ground Camera Network
- SeCap: Self-Calibrating and Adaptive Prompts for Cross-view Person Re-Identification in Aerial-Ground Networks
- View Confusion Feature Learning for Person Re-Identification
- SD-ReID: View-aware Stable Diffusion for Aerial-Ground Person Re-Identification
- LATex: Leveraging Attribute-based Text Knowledge for Aerial-Ground Person Re-Identification
- AG-ReID.v2: Bridging Aerial and Ground Views for Person Re-Identification
- Deep Pre-Alignment for VLMs