Joint Attentive Spatial-Temporal Feature Aggregation for Video-Based Person Re-Identification

Lin Chen, Hua Yang, Zhiyong GaoIEEE Access 2019 · OpenReview

Abstract: Video-based person re-identification (Re-ID) remains to be a promising but challenging computer vision task, suffering from a lack of discriminative features that better aggregate both the spatial and temporal information. In this paper, we propose a joint attentive spatial-temporal feature aggregation network (JAFN) for the video-based person Re-ID, simultaneously learning the quality- and frame-aware model to obtain attention-based spatial-temporal feature aggregation. Specifically, we utilize CNN to learn the spatial features, while introducing the LSTM to separately learn the temporal features. For the feature aggregation, we introduce two attention mechanisms respectively for generating the quality and frame significance score, where the quality score measures the quality of the images for attentive spatial feature aggregation, and the frame score measures the significance of the image frames contributing to the temporal feature. Then, we utilize the set-pooling for both the quality-aware spatial feature and the frame-aware temporal feature aggregation based on the attentive scores. The residual learning is also introduced to play between the LSTM and the CNN for adaptive spatial-temporal feature fusion. Furthermore, we adopt the data balance to alleviate the data disproportions existing in datasets of the video-based Re-ID. The extensive experimental results conducted on the PRID2011, i-LIDS-VID, and MARS datasets demonstrate the effectiveness of the proposed JAFN. Furthermore, comparison results conducted on different modules and features in the JAFN show that our approach is of favorable generalization ability on attentively aggregating both the spatial and temporal features.

Introduction

Person re-identification (Re-ID) is the critical task of identifying the same individual across non-overlapping surveillance camera views. While image-based Re-ID relies on static appearance, video-based Re-ID leverages richer temporal data, such as gait and movement patterns. However, video streams are often noisy, containing blurry frames, occlusions, or poor viewpoints that can degrade performance if treated with equal importance.

The proposed Joint Attentive Spatial-Temporal Feature Aggregation Network (JAFN) addresses this by intelligently filtering and fusing spatial and temporal information. By selectively weighting the most informative images and frames, JAFN creates a more discriminative feature representation for robust identification.

The JAFN Framework

The core of JAFN lies in its ability to simultaneously learn quality-aware and frame-aware models for optimized feature aggregation.

Overview of the JAFN approach, showing how video sequences are processed through quality-aware and frame-aware attention modules.

Dual Attention Mechanisms

  1. Quality-aware Attention: This CNN-based module evaluates the visual clarity of each image. Frames that are clear and unobstructed receive higher "quality scores," ensuring they contribute more significantly to the final spatial feature representation.
  2. Frame-aware Attention: Utilizing an LSTM-based module, the network assesses the temporal significance of frames within a sequence. This allows the model to prioritize frames that best capture distinctive movement patterns, like a person's specific gait.

Residual Feature Fusion

Rather than simple concatenation, JAFN employs a residual learning framework to fuse spatial and temporal data. Temporal features from the LSTM are treated as a residual component added to the spatial features from the CNN. This adaptive approach allows the network to focus on the complementary information provided by movement patterns.

Detailed architecture of the JAFN network, highlighting the dual-stream processing for spatial (FCN) and temporal (LSTM) features with residual fusion.

Optimization and Data Balancing

To handle the data disproportions common in Re-ID datasets, the authors implement a data balancing strategy. The model is trained using a multi-loss objective—combining Siamese, Triplet, and Softmax losses—to ensure that features of the same identity are clustered together while distinct identities are clearly separated in the feature space.

Experimental Evaluation

The JAFN model was rigorously tested on three benchmark datasets, demonstrating significant gains over baseline models:

  • PRID-2011: A clean outdoor dataset where JAFN achieved a 97.4% Rank-1 accuracy, a 7% improvement over the baseline.
  • i-LIDS-VID: A challenging airport dataset with heavy occlusion. JAFN reached 76.0% Rank-1 accuracy, with temporal information providing an 8% boost over spatial-only methods.
  • MARS: The largest benchmark, featuring over 1,200 identities. JAFN achieved 89.3% Rank-1 accuracy and a Mean Average Precision (mAP) of 74.9%.

Key Insights

  • Selective Aggregation: The dual-attention mechanism effectively filters out "noise" from low-quality frames, which is crucial for real-world surveillance where conditions are rarely ideal.
  • Complementary Strengths: While spatial appearance is the primary driver for identification, temporal features act as a vital "safety net" in environments with drastic lighting or viewpoint changes.
  • Efficiency of Residuals: The residual learning structure simplifies the optimization process, allowing the model to efficiently learn the "added value" of temporal data.

Conclusion

JAFN demonstrates that joint attention to both image quality and temporal significance significantly enhances video-based person Re-ID. While performance in extremely cluttered environments like i-LIDS-VID remains a challenge, the framework's ability to adaptively fuse spatial and temporal features sets a strong foundation for future work. Potential improvements include exploring more advanced backbones like Transformers or more sophisticated synthetic data generation techniques to further address dataset imbalances.

Summary updated

Abstract

Video-based person re-identification (Re-ID) remains to be a promising but challenging computer vision task, suffering from a lack of discriminative features that better aggregate both the spatial and temporal information. In this paper, we propose a joint attentive spatial-temporal feature aggregation network (JAFN) for the video-based person Re-ID, simultaneously learning the quality- and frame-aware model to obtain attention-based spatial-temporal feature aggregation. Specifically, we utilize CNN to learn the spatial features, while introducing the LSTM to separately learn the temporal features. For the feature aggregation, we introduce two attention mechanisms respectively for generating the quality and frame significance score, where the quality score measures the quality of the images for attentive spatial feature aggregation, and the frame score measures the significance of the image frames contributing to the temporal feature. Then, we utilize the set-pooling for both the quality-aware spatial feature and the frame-aware temporal feature aggregation based on the attentive scores. The residual learning is also introduced to play between the LSTM and the CNN for adaptive spatial-temporal feature fusion. Furthermore, we adopt the data balance to alleviate the data disproportions existing in datasets of the video-based Re-ID. The extensive experimental results conducted on the PRID2011, i-LIDS-VID, and MARS datasets demonstrate the effectiveness of the proposed JAFN. Furthermore, comparison results conducted on different modules and features in the JAFN show that our approach is of favorable generalization ability on attentively aggregating both the spatial and temporal features.
BibTeX
@inproceedings{chen2019joint,
  author = {Lin Chen and Hua Yang and Zhiyong Gao},
  title = {{Joint Attentive Spatial-Temporal Feature Aggregation for Video-Based Person Re-Identification}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=AWn6WyHHip},
}
Figure