An Attention-Enhanced Network for Person Re-Identification via Appearance–Gait Fusion

Zelong Yu, Yiyu Cai, Xu Hanming, Lei Chen, Mingqian Yang, Huabo Sun, Xiangyu ZhaoElectronics 2025 · DOI

TL;DR

This paper introduces an attention-enhanced network that fuses appearance and gait information to improve person re-identification (Re-ID) accuracy, especially under varying camera viewpoints. By using a novel multimodal information exchange module and perspective-aware weighting, their method achieves state-of-the-art results on the challenging CASIA-B dataset, demonstrating increased accuracy and stability.

What Problem Does This Paper Address?

Person re-identification (Re-ID) is crucial for security systems, aiming to identify individuals across different surveillance cameras. However, a major challenge is the significant drop in accuracy caused by variations in camera viewpoints. Traditional Re-ID methods often struggle with these perspective changes, leading to unreliable identification. While multimodal approaches combining appearance (e.g., from RGB images) and gait (e.g., from Gait Energy Images or GEIs) have shown promise, they haven't fully addressed the impact of viewpoint variations. This paper tackles this problem by proposing a method that explicitly leverages the complementary information from appearance and gait, guided by attention mechanisms and perspective awareness, to create a more robust Re-ID system.

Methods

The core of the proposed method is an attention-enhanced network that fuses appearance and gait features.

  • Dual-Branch Feature Extraction: Two ResNet-50 networks are used to extract features from RGB images (appearance) and Gait Energy Images (GEIs) (gait) independently.
  • Multimodal Information Exchange (MIE) Module: This is the key innovation. It's designed to facilitate deep interaction and mutual guidance between the appearance and gait feature streams during the feature extraction process. This module uses an adapted attention mechanism (based on AAOE) that enhances, rather than weakens, feature learning. It's applied at two different layers of the ResNet-50 backbone (layer2 and layer4) to capture interactions at both shallow and deep feature levels. The module uses element-wise addition for feature fusion, which the authors found to be most effective at retaining information from both modalities.
  • Importance Weight Estimation: To further account for perspective variations, the system estimates the pedestrian's viewpoint angle. This angle is then used to generate importance weights for the appearance and gait features. This ensures that features more relevant to a given perspective are emphasized.
  • Autoencoder-based Fusion: Finally, the weighted appearance and gait features are fused using an autoencoder. This module refines and simplifies the combined features, removing redundancy while preserving crucial recognition information.

Datasets

The primary dataset used for training and evaluation is CASIA-B. This dataset is well-known for its gait recognition challenges, featuring 124 pedestrians observed from 11 different viewpoints (0° to 180°) and under various clothing conditions (normal walking, with a coat, and with a backpack). For this study, only the "Normal Walking" (NM) sequences were used to isolate the effect of viewpoint variations.

Key Insights and Ideas

  • Synergy of Appearance and Gait: The paper strongly advocates for the combined use of appearance and gait features, highlighting that they offer complementary information that is essential for robust Re-ID, especially when one modality might be less informative due to viewpoint changes.
  • Attention for Cross-Modal Guidance: The MIE module's success demonstrates that attention mechanisms are not just for intra-modal feature enhancement but can be powerful tools for guiding and enriching cross-modal interactions.
  • Layered Interaction is Crucial: Applying the MIE module at multiple layers (both shallow and deep) significantly boosts performance, suggesting that interactions at different levels of feature abstraction are vital for capturing complex relationships.
  • Perspective-Aware Weighting: Explicitly mapping viewpoint angles to feature importance weights is an effective strategy to mitigate the negative impact of perspective variations.
  • Element-wise Addition for Fusion: Surprisingly, simple element-wise addition proved more effective than multiplication or concatenation for fusing features within the MIE module, likely due to its ability to preserve information from both modalities without suppression.

Key Results

  • State-of-the-Art Performance: The proposed method achieved a mean accuracy of 94.9% on the CASIA-B dataset under viewpoint variation, which is 1.1% higher than the next best method.
  • Improved Stability: The method also achieved the smallest variance of 4.199, indicating significantly better robustness and stability across different viewpoints compared to existing methods.
  • Cross-Domain Generalization: Even when tested on sequences with different clothing (backpacks), the method showed strong performance (87.3% mean accuracy) and low variance (27.43), outperforming other methods under domain shift.
  • Ablation Studies: Experiments confirmed the effectiveness of the MIE module, the importance of its layered application (layer2 and layer4), and the superiority of the proposed AAOE attention mechanism.

Limitations and Future Work

  • Appearance-Centric Weighting: The authors acknowledge that the current importance weighting estimation primarily focuses on appearance features. This can lead to failures when pedestrian appearances are very similar (e.g., identical clothing). Future work could focus on enhancing the model's ability to discern subtle appearance differences and further strengthening gait feature expressiveness.
  • Handling Other Challenges: The paper suggests that future research should also consider factors like illumination and occlusion, which are common in real-world surveillance scenarios.
  • Diverse Modalities and Lightweight Models: Exploring the integration of other image types (e.g., infrared, low-light) and developing more efficient, lightweight models to reduce computational costs are also identified as promising future directions.

Summary updated

Abstract

The objective of person re-identification (Re-ID) is to recognize a given target pedestrian across different cameras. However, perspective variations, resulting from differences in shooting angles, often significantly impact the accuracy of person Re-ID. To address this issue, this paper presents an attention-enhanced person Re-ID algorithm based on appearance–gait information interaction. Specifically, appearance features and gait features are first extracted from RGB images and gait energy images (GEIs), respectively, using two ResNet-50 networks. Then, a multimodal information exchange module based on the attention mechanism is designed to build a bridge for information exchange between the two modalities during the feature extraction process. This module aims to enhance the feature extraction ability through mutual guidance and reinforcement between the two modalities, thereby improving the model’s effectiveness in integrating the two types of modal information. Subsequently, to further balance the signal-to-noise ratio, importance weight estimation is employed to map perspective information into the importance weights of the two features. Finally, based on the autoencoder structure, the two features are weighted and fused under the guidance of importance weights to generate fused features that are robust to perspective changes. The experimental results on the CASIA-B dataset indicate that, under conditions of viewpoint variation, the method proposed in this paper achieved an average accuracy of 94.9%, which is 1.1% higher than the next best method, and obtained the smallest variance of 4.199, suggesting that the method proposed in this paper is not only more accurate but also more stable.
BibTeX
@inproceedings{yu2025attention,
  author = {Zelong Yu and Yiyu Cai and Xu Hanming and Lei Chen and Mingqian Yang and Huabo Sun and Xiangyu Zhao},
  title = {{An Attention-Enhanced Network for Person Re-Identification via Appearance–Gait Fusion}},
  booktitle = {Electronics},
  year = {2025},
  doi = {10.3390/electronics14214142},
}
Figure