Reading People Like Cryo-EM Particles: Graph Autodecoders for Aerial Re-Identification

Matching a person seen by a street camera to the same person filmed by a drone at 120 meters is one of the hardest problems in surveillance vision. From above, the body is a few dozen pixels, heavily foreshortened, and often in different clothes. Methods that fit a full 3D body mesh to each frame break down exactly here. A recent aerial-ground benchmark reports a 27% drop in top-1 matching accuracy as altitude rises from 15 m to 120 m (AG-VPReID). Structural biology has already solved a problem with the same shape. Cryo-electron microscopy (cryo-EM) recovers a protein's 3D structure from hundreds of thousands of extremely noisy 2D images. Each image shows the molecule from an unknown angle and in a slightly different shape. The solution is to fit one shared 3D model to all the images at once, not to reconstruct each image separately.

We propose to treat a person's tracklet, meaning the sequence of detections of one person over time, the way cryo-EM treats a particle stack. The person becomes a molecule-like graph. Joints are atoms and bones are bonds. Bone lengths play the role of bond lengths: they stay fixed for the identity. Joint angles play the role of conformational change: they vary per frame. Each frame's camera view is unknown and is estimated during fitting. A graph-network decoder is fit to every weak 2D keypoint detection in the tracklet through a differentiable camera projection. The identity code that comes out is a set of skeletal proportions that, by construction, does not depend on viewpoint, pose, or clothing.

Core Claim

Recovering identity geometry jointly across many low-resolution views should beat estimating 3D geometry frame by frame and averaging. It should also beat it most where per-frame methods fail, at high altitude. The specific new ingredient is a decoder built so that bone lengths can depend only on a per-identity code and joint angles only on a per-frame code. This takes the bond-graph and bond-angle-graph split from molecular representation learning and turns it into an identity/pose split. The decoder is then fit to the observations the way a cryo-EM reconstruction fits a structure to its particle images.

Prior Work and Positioning

The method comes mainly from Protein Graph Neural Networks for Heterogeneous Cryo-EM Reconstruction. That paper fits a graph-neural-network "autodecoder" (a decoder whose per-image input codes are learned directly by optimization, with no encoder) to noisy projections. The decoder deforms a template protein backbone, and the paper estimates unknown orientations jointly through Ellipsoidal Support Lifting. Its graph decoder beats an MLP decoder: 1.09 Å versus 1.24 Å RMSD on adenylate kinase and 2.08 Å versus 2.37 Å on NSP-13. The authors also observe that message passing over bonds acts as an implicit regularizer against physically impossible shapes. We reuse this recipe but change three things.

First, the domain is human skeletons under weak perspective projection rather than electron projections with a microscope contrast transfer function. Second, we split the latent into a shared identity code and per-frame pose codes. Krook et al. have one code per image and do not separate invariant from variable geometry. Third, the output is a retrieval embedding, not a reconstruction.

The decoder's structure comes from GEM, which passes messages between an atom-bond graph and a bond-angle graph so that angles become primary features. We assign the bone graph to identity and the bone-angle graph to pose. GEM itself works only on precomputed 3D conformers and has no projection model. Cormorant builds exact rotation equivariance into molecular networks. We do not adopt its expensive tensor products, because the rotation here is a camera rotation estimated explicitly in the projection, which makes the identity code rotation-invariant by construction.

On the vision side, three works motivate the proposal. 3DInvarReID separates a naked-body identity code from clothing through implicit occupancy fields and RGB re-rendering. It depends on an off-the-shelf pose estimator, needs pixels detailed enough to render, and has no relational graph. SMPLGait normalizes silhouette features using per-frame SMPL parameters (the parameters of a standard parametric 3D body model). It reaches 53.20% top-1 accuracy on Gait3D versus 42.60% for GaitSet, and loses more than 5% without its 3D branch, which is evidence that 3D helps. CSCL maps pixels to body-model vertices, which requires dense pixels.

Two works constrain the design. AG-VPReID shows that per-frame mesh recovery collapses at high altitude. Graph re-identification models such as PGGANet build graphs over 2D keypoints and stay planar, so they inherit projective distortion. Our graph lives in 3D and only its projection is compared to the image.

Research Program

For identity $k$ with tracklet frames $t = 1..T$, let $u_{tj}$ be the detected 2D location of joint $j$ and $c_{tj}$ its detector confidence. The decoder $D_G$ is a two-stream graph network on the skeleton graph $G$. It takes an identity code $z_k$, which is passed only through bone-node updates that output bone-length ratios. It also takes a frame code $w_t$, which is passed only through bone-angle-graph updates that output joint rotations. Forward kinematics combines the two into 3D joints. A per-frame weak-perspective camera $\Pi_t$ (rotation, scale, offset) projects them. We fit

$$ \min_{z_k,\{w_t,\Pi_t\}} \sum_{t=1}^{T}\sum_{j} c_{tj}\,\big\| u_{tj} - \Pi_t\big(D_G(z_k, w_t)\big)_j \big\|^2 + \lambda \sum_t \|w_t\|^2 . $$

The last term is a small penalty that keeps the per-frame pose codes from absorbing structure that belongs in the identity code. Weak perspective makes absolute scale unrecoverable, so identity is defined as bone-length ratios. This is the same quantity anthropometric invariance relies on.

Camera rotation is initialized by a coarse multi-hypothesis search, which mirrors the role of ESL. Drone tracklets naturally span many viewing angles, and that spread is what makes the depth of each bone recoverable. A bone that is foreshortened in one frame appears at full length in another.

The decoder weights are trained on Gait3D, which provides 2D and 3D keypoints and SMPL fits, together with an identity-contrastive loss on $z_k$. At test time, $z_k$ is obtained by per-tracklet optimization. For speed, we also distill an amortized encoder from the optimized codes and report both versions. The geometric code is fused with an appearance model through reciprocal rank fusion, as in AG-VPReID-Net. The working hypothesis is that the geometric code adds most where appearance fails: at high altitude, under clothing changes, and in uniform-scrub settings like those motivating TrackOR.

Baselines

The naive reference is appearance alone, meaning AG-VPReID-Net's appearance streams without any geometric code. This measures what the geometric code adds at all.

The strongest existing method is AG-VPReID-Net (AG-VPReID), which combines CLIP features, SMPL mesh recovery, and UV texture normalization. It reports 75.6% top-1 for ground-to-aerial retrieval. We will retrain it with its released protocol on the same splits. We will fuse our code into it with the same reciprocal rank fusion and the same tuning budget. Beating it means a top-1 gain for the fused system that is largest in the 80 m and 120 m subsets and statistically clear across three seeds. The headline target is to cut the 15-to-120 m drop below 27%.

On Gait3D, the strongest baseline is SMPLGait at 53.20% top-1. We compare our code fused with GaitSet features against it on identical splits.

Two ablations isolate the key ingredient. The first, the averaged-shape baseline, takes the per-frame SMPL shape parameters from the same mesh recovery network and averages them over the tracklet. This tests whether joint multi-view fitting matters beyond pooling. The second replaces the graph decoder with an MLP decoder under the identical objective and budget. This directly repeats Krook et al.'s GNN-versus-MLP comparison in the human setting. A third, smaller ablation removes the identity/pose routing constraint so that both codes can affect bone lengths.

Why not just…?

Why not just average SMPL shape parameters over the tracklet? This is the most serious threat, since body-model shape already encodes proportions. Per-frame shape regression is a 2D-to-3D guess made from tens of pixels, and AG-VPReID's altitude drop indicates that its errors are large and correlated with viewpoint. Averaging does not remove an error that is biased by viewpoint. Joint fitting does, because it requires one skeleton to explain every view. The averaged-shape ablation settles this directly. If it matches our method at 120 m, the joint-fitting claim is wrong, and we will report it that way.

Why not just use a CLIP-based clothing-agnostic model? CCAF reaches 70.4% top-1 on PRCC without 3D modeling. However, it depends on a human-parsing network that fails under unusual viewpoints, and AG-VPReID reports that CLIP-ReID degrades under aerial foreshortening. We treat it as complementary appearance input, not a competitor. The fused-versus-appearance-only comparison will show whether geometry adds anything on top of it.

Why not just a planar 2D pose graph? Models like PGGANet compare keypoint graphs in the image plane, where bone lengths change with viewing angle. Gait3D's silhouette results, GaitSet at 42.60% versus SMPLGait at 53.20%, suggest that planar cues discard the out-of-plane information this task needs. We include a 2D keypoint-graph encoder trained on the same detections as a reference.

Proposed Experiments

The first decisive check is controlled and runs before any retrieval. We render Gait3D's 3D skeletons at simulated drone altitudes, add keypoint noise and drop out low-confidence joints, then measure bone-ratio recovery error against ground truth as a function of tracklet length and the spread of viewing angles. For comparison we run the averaged-shape and MLP-decoder ablations. If the identity code is genuinely pooling independent views, error should fall roughly with the square root of the number of distinct views, as when averaging independent noisy measurements. If error stops improving once views are similar, the gain comes from angular diversity rather than from frame count, and we would report that distinction.

The second check is retrieval on AG-VPReID, stratified by altitude. We also evaluate on Gait3D against SMPLGait's 53.20%, and on the UAV-Human re-identification split (sutdcv/UAV-Human), for which no reported re-identification result was found on Lacuna. The third check is a homogeneous-appearance stress test: we evaluate on clothing-changing PRCC subsets as a proxy for uniform scrubs, since no reported TrackOR number was found on Lacuna. Throughout, we report the wall-clock cost of test-time optimization against the amortized encoder, because a geometric code that needs minutes per tracklet answers a different deployment question than one that needs milliseconds.