k-Reciprocal Harmonious Attention Network for Video-Based Person Re-Identification

Xinxing Su, Xiaoye Qu, Zhikang Zou, Pan Zhou, Wei Wei, Shiping Wen, Menglan Hu

Introduction

Person re-identification (re-ID) involves recognizing individuals across non-overlapping camera views. While image-based re-ID is common, video-based re-ID offers richer temporal cues like gait and motion. However, video sequences often suffer from motion blur, redundant frames, and frequent occlusions. Traditional approaches using Recurrent Neural Networks (RNNs) or LSTMs often struggle to capture long-range dependencies and can be difficult to optimize with limited training data. The k-Reciprocal Harmonious Attention Network (KHAN) addresses these challenges by extracting discriminative features that are "harmonious" across space, channel, and time.

Methodology

KHAN is built upon a ResNet-50 backbone and introduces two primary novel components to handle the complexities of video data.

Harmonious Attention Module (HAM)

HAM utilizes "non-local" attention mechanisms. Instead of focusing solely on local pixel neighborhoods, it computes interactions between every pair of spatial positions and every pair of feature channels. This allows the network to learn global relationships, such as associating a specific shirt color with pants regardless of their exact position in the frame.

k-Reciprocal Attention Module (KAM)

KAM is the core innovation of this work. Rather than using a "black box" LSTM for temporal summarization, KAM employs a non-parametric discriminative feature selection algorithm. It identifies "key features" by finding those that appear frequently within each other’s $k$-reciprocal nearest neighbor sets across the video timeline. This process effectively filters out noise from blurry or occluded frames, creating a more robust representation of the pedestrian.

Illustration of (a) temporal k-reciprocal mining and (b) spatial/channel harmonious attention in the KHAN architecture.

The model is trained using a combination of Batch-Hard Triplet Loss to improve feature clustering and Softmax Cross-Entropy Loss for identity classification.

Experimental Evaluation

The authors evaluated KHAN on three standard video re-ID benchmarks, achieving state-of-the-art (SOTA) results across the board:

  • MARS: On this large-scale dataset, KHAN reached 85.7% Rank-1 accuracy and 77.8% mAP, significantly outperforming previous methods like DRSTAN.
  • iLIDS-VID: In this occlusion-heavy environment, the model achieved 82.3% Rank-1, surpassing the previous SOTA by over 2%.
  • PRID2011: KHAN achieved a near-perfect 93.7% Rank-1 accuracy despite significant lighting changes between cameras.

Ablation studies confirmed that the synergy between HAM and KAM is essential; removing either module led to a noticeable drop in performance, proving that spatial, channel, and temporal attention must work in concert.

Conclusion

KHAN demonstrates that a non-parametric approach to temporal attention, specifically through $k$-reciprocal mining, can be more effective and efficient than traditional recurrent architectures for video-based re-ID. By balancing global spatial relationships with robust temporal feature selection, the network successfully mitigates the impact of occlusions and noise. Future research directions include making the $k$-reciprocal parameters adaptive and integrating the system into real-time online pedestrian tracking applications.

Summary updated

Abstract

Video-based person re-identification aims to retrieve video sequences of the same person in the multi-camera system. In this paper, we propose a k -reciprocal harmonious attention network (KHAN) to jointly learn discriminative spatiotemporal features and the similarity metrics. In KHAN, the harmonious attention module adaptively calibrates response at each spatial position and each channel by explicitly inspecting position-wise and channel-wise interactions over feature maps. Besides, the k-reciprocal attention module attends key features from all frame-level features with a discriminative feature selection algorithm; thus, useful temporal information within contextualized key features can be assimilated to produce more robust clip-level representation. Compared with commonly used local-context based approaches, the proposed KHAN captures long dependency of different spatial regions and visual patterns while incorporating informative context at each time-step in a non-parametric manner. The extensive experiments on three public benchmark datasets show that the performance of our proposed approach outperforms the state-of-the-art methods.
BibTeX
@article{su2019k,
  author = {Xinxing Su and Xiaoye Qu and Zhikang Zou and Pan Zhou and Wei Wei and Shiping Wen and Menglan Hu},
  title = {{k-Reciprocal Harmonious Attention Network for Video-Based Person Re-Identification}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2898269},
}
Figure