Traditional person re-identification (Re-ID) systems rely heavily on superficial appearance cues, such as the color and texture of a person's clothing. While highly descriptive in the short term, these features are fundamentally volatile. When an individual changes their coat, carries a bag, or moves across cameras with varying lighting conditions, purely appearance-based models routinely fail.
To overcome this limitation, a prominent research direction has emerged: multimodal biometric fusion. Instead of relying on transient visual textures, modern systems combine static structural biometrics (such as anthropometric measurements and 3D body shapes) with dynamic behavioral biometrics (such as gait patterns and skeletal joint trajectories). By anchoring volatile visual data to invariant physical structures and movement mechanics, these systems achieve robust, long-term identification.
The Volatility of Appearance
The core vulnerability of standard Re-ID models is systematically exposed in long-term tracking scenarios. As demonstrated by Fangbin Wan et al. in When Person Re-identification Meets Changing Clothes (cvpr 2020) and Peng Xu et al. in DeepChange: A Long-Term Person Re-Identification Benchmark with Clothes Change (iccv 2023), a target's visual profile changes drastically across seasons and outfits. When an individual removes a jacket or alters their accessories, whole-body tracking systems experience immediate continuity failures (Kunihiro Moriga et al. in Creating Value from Deep Learning Technology and Its Business Applications (NTT 2019)).
Furthermore, Jiahao Nie et al. note in Color Space Learning for Cross-Color Person Re-Identification (arxiv 2024) that appearance models are fundamentally bounded when color information becomes deceptive or missing entirely, such as in low-resolution or infrared surveillance. The solution is not to engineer better color descriptors, but to introduce a complementary biometric safety net: the geometry of the human body and the mechanics of how it moves.
The Complementary Nature of Shape and Motion
Why does fusing shape and motion work so effectively? The intuition lies in their statistical independence. Two individuals might share an identical physical build or height, yet their walking styles—the trajectories of their joints and their stride frequencies—are often distinct.
Early foundational work by Liang Wang et al. in Fusion of Static and Dynamic Body Biometrics for Gait Recognition (iccv 2003) proved that combining static physical build with dynamic joint motion at the decision level yields a massive jump in accuracy because the two modalities capture independent information.
This relationship is highly symmetrical:
- When motion is unavailable: If a subject is standing still or sitting, motion-based signatures fail. As explored by Ryo Yonetani et al. in Ego-Surfing: Person Localization in First-Person Videos Using Ego-Motion Signatures (IEEE Trans. Pattern Anal. Mach. Intell. 2018), relying purely on motion signatures creates vulnerabilities during periods of minimal activity. Fusing these signatures with static visual features stabilizes the identity.
- When appearance is obscured: Conversely, when a subject changes clothing, their underlying bone structure remains constant. The authors of Person Identification Using Anthropometric and Gait Data from Kinect Sensor (aaai 2015) demonstrated that static anthropometric measurements (such as limb lengths and height) act as reliable anchors that compensate for variations in walking style, creating a highly resilient hybrid signature.
Disentangling Motion from Silhouette Distortions
In computer vision, gait is traditionally extracted using silhouettes—binary masks that isolate the walking human form from the background. However, silhouettes are highly sensitive to "covariates" like heavy winter coats, flowing dresses, or carried backpacks, which artificially distort the perceived body shape.
To solve this, researchers use skeletal keypoint data to regularize and guide appearance features.
[Volatile Input Video]
│
├──► Silhouette Stream (Captures Shape, but distorted by clothing/bags)
│ ▲
│ │ Cross-Modal Guidance / Attention
│ ▼
└──► Skeleton Stream (Captures Pure Joint Mechanics, clothing-invariant)
│
▼
[Fused Identity Embedding]
For example, Yufeng Cui et al. developed MMGaitFormer in Multi-modal Gait Recognition via Effective Spatial-Temporal Feature Fusion (cvpr 2023), which fuses silhouette-based appearance features with graph-based skeleton pose data. Because the skeleton graph tracks internal joint locations, it serves as a stable structural reference frame. If a bulky coat alters the outer silhouette contour, the skeleton stream ensures the model still prioritizes the underlying motion dynamics.
Similarly, Hung-Min Hsu et al. proposed GaitTAKE in GaitTAKE: Gait Recognition by Temporal Attention and Keypoint-guided Embedding (arxiv 2022), utilizing a learned temporal attention mechanism. By leveraging skeletal stability, the network learns to focus on the specific frames in a walking sequence where the gait mechanics are most visible and least obscured by external clothing.
This cross-modal interaction is bidirectional. In GaitMA: Pose-guided Multi-modal Feature Fusion for Gait Recognition (arxiv 2024), Fanxu Min et al. show that skeletal joint heatmaps can "fill in the blanks" when silhouettes are degraded by complex backgrounds or self-occlusion. Conversely, Haidong Zhu et al. demonstrate in GaitRef: Gait Recognition with Refined Sequential Skeletons (arxiv 2023) that the temporal consistency of silhouettes can be used to filter out noise and jitter from raw skeleton data, refining the accuracy of the behavioral signature.
Geometric Invariance via Relative Metrics
A major hurdle in gait analysis is viewpoint and distance variation; a person walking toward a camera projects a completely different visual trajectory than someone walking perpendicularly. To achieve true geometric invariance, models must decouple pure mechanics from absolute visual coordinates.
In Two-Point Gait: Decoupling Gait from Body Shape (iccv 2013), Stephen Lombardi et al. introduced a method to extract a "pure" motion representation based on relative pixel displacements, entirely independent of morphological shape. This pure motion is then fused back with static shape descriptors to maximize identification performance.
Taking this concept further, researchers began utilizing 3D sensors to extract view-invariant geometric features. Yong Dou et al. in Relative distance features for gait recognition with Kinect (arxiv 2016) and Fei Zhang et al. in Exploring Relative Motion Features for Gait Recognition with Kinect (ICONIP (4) 2015) demonstrated that tracking the relative distances between joints (such as the changing distance between ankles or elbows over time) provides a highly accurate behavioral signature. Because these metrics rely on internal ratios rather than absolute spatial coordinates, they remain invariant to camera distance and perspective, while remaining lightweight enough for real-time processing.
Advanced Fusion Architectures
How do modern networks mathematically bridge the gap between these heterogeneous streams? Several sophisticated fusion mechanisms have been developed:
- Sequence Masking: Zhigang Chang et al. developed Seq-Masks in Seq-Masks: Bridging the gap between appearance and gait modeling for video-based person re-identification (arxiv 2021), which uses sequential masks to explicitly map the transition between static visual data and dynamic gait sequences across camera networks.
- Perspective-Aware Attention: In An Attention-Enhanced Network for Person Re-Identification via Appearance–Gait Fusion (Electronics 2025), Zelong Yu et al. introduce a multimodal information exchange module that dynamically weights appearance versus gait based on the camera's viewpoint angle, optimizing which feature dominates the final embedding.
- Cascade Ranking: Lan Wei et al. proposed a tournament-style model in Swiss-System Based Cascade Ranking for Gait-Based Person Re-Identification (aaai 2015) that iteratively groups and ranks candidates against similar competitors, effectively combining multiple distinct gait and shape features in stages.
- 3D Mesh Reconstruction: To handle unconstrained "in-the-wild" scenarios where subjects might be completely stationary, Haidong Zhu et al. introduced ShARc in ShARc: Shape and Appearance Recognition for Person Identification In-the-wild (wacv 2024). This framework integrates 2D silhouettes and skeleton motion with full 3D body mesh reconstruction, ensuring a reliable biometric safety net even when motion cues are absent.
- Heterogeneous Non-Visual Fusion: The utility of this paradigm extends beyond surveillance. For instance, Lei Jin et al. developed E3SFormer in Vision-based multimodal energy expenditure estimation for aerobic exercise in adults (Frontiers in Physiology 2025), demonstrating that fusing dynamic vision sequences with static physical attributes (age, weight) and physiological signals via multi-layer perceptrons allows models to highly personalize their estimations.
Ultimately, as highlighted by Saihui Hou et al. in An In-Depth Exploration of Person Re-Identification and Gait Recognition in Cloth-Changing Conditions (cvpr 2023), Yan Huang et al. in Attribute-Guided Pedestrian Retrieval: Bridging Person Re-ID with Internal Attribute Variability (cvpr 2024), and the authors of Person Re-Identification by Contour Sketch Under Moderate Clothing Change (IEEE Transactions on Pattern Analysis and Machine Intelligence 2019), the trajectory of the field is moving toward deep cross-modal distillation. By forcing networks to learn the underlying relationships between volatile outer textures and invariant inner mechanics, we can train computer vision models to look past superficial clothing cues and recognize the immutable physical signatures underneath.
Key Foundational Work
Fusion of Static and Dynamic Body Biometrics for Gait Recognition (2003) — contributed 1 concept elements to this direction.
Person Identification Using Full-Body Motion and Anthropometric Biometrics from Kinect Videos (2012) — contributed 1 concept elements to this direction.