A Cross-Modal Distillation Network for Person Re-identification in RGB-Depth

Frank Hafner, Amran Bhuiyan, Julian F. P. Kooij, Eric Granger

Introduction

Person re-identification (re-ID) is the task of recognizing the same individual across a network of distributed cameras, serving as a cornerstone for modern surveillance, autonomous driving, and robotics. While single-modality re-ID (e.g., RGB-to-RGB) has seen significant progress, matching across different sensor modalities—specifically RGB to Depth (RGB-D)—remains a formidable challenge.

The "modality gap" is the primary hurdle: the data distribution of color pixels is fundamentally different from that of a depth map. Standard re-ID difficulties like occlusions, varying viewpoints, and pose changes further complicate this gap. As autonomous systems increasingly integrate LiDAR/depth sensors alongside standard cameras, bridging the RGB-Depth gap is becoming a critical real-world requirement.

Conceptual comparison between single-modal and cross-modal re-identification.

Proposed Methodology

To address the modality gap, this research introduces a Cross-Modal Distillation training procedure combined with a Cross-Modal Gated Attention network.

Two-Step Distillation

Rather than training on both modalities simultaneously, the authors employ a teacher-student approach:

  1. Step I: A standard re-ID network (ResNet50) is trained on the first modality (e.g., Depth) using Triplet and Softmax losses.
  2. Step II: A second network for the target modality (e.g., RGB) is initialized with the weights from Step I. Mid-level layers are frozen to preserve high-level structural representations. The network is then trained using a Mean Squared Error (MSE) loss, forcing the embeddings of paired RGB-Depth images into the same latent space.
The two-step distillation training process and the final cross-modal inference system.

Cross-Modal Gated Attention

To mitigate spatial misalignment, the system uses features from one modality to generate a spatial soft attention map. This map acts as a gating signal for the backbone features of the other modality during training. During inference, the network performs self-attention when only one modality is available.

Experimental Evaluation

The method was evaluated on two public RGB-D datasets captured with Microsoft Kinect cameras:

  • BIWI RGBD-ID: Focuses on long-term re-ID with ~22,000 images across 78 instances.
  • RobotPKU: Features ~16,500 images of 90 persons, characterized by noisier depth data and more dynamic movements.

Key Insights

  • Directional Transfer: Distilling knowledge from Depth to RGB is significantly more effective than the reverse. Depth maps emphasize robust structural and shape features that are also present in RGB, whereas RGB relies on color information absent in depth data.
  • Structural Focus: Visualizations show that while standard RGB networks focus on clothing color, the distillation process shifts the network's focus toward physical structure and body shape, aligning it with the depth modality.
  • Embedding Optimization: Performance is sensitive to embedding size; optimal sizes (e.g., 512 for BIWI, 256 for RobotPKU) provided noticeable improvements.

Results

The proposed architecture outperforms existing state-of-the-art (SOTA) cross-modal methods by up to 16.1% in mean Average Precision (mAP).

Dataset mAP Rank-1 Accuracy
BIWI 41.3% 40.4%
RobotPKU 23.5% 25.3%

Ablation studies highlighted the importance of weight copying and layer freezing; omitting these steps led to a drop in mAP of up to 18%, confirming the necessity of strict structural guidance.

Conclusion and Future Work

While this method sets a new benchmark for cross-modal RGB-D re-ID, absolute performance (approx. 40% Rank-1) still lags behind single-modal RGB re-ID (often >90%). This gap underscores the complexity of the task. Future research should focus on "in-the-wild" datasets beyond controlled lab environments and the development of larger RGB-D datasets to support more robust deep learning architectures.

Summary updated

BibTeX
@article{hafner2022cross,
  author = {Frank Hafner and Amran Bhuiyan and Julian F. P. Kooij and Eric Granger},
  title = {{A Cross-Modal Distillation Network for Person Re-identification in RGB-Depth}},
  journal = {Comput. Vis. Image Underst.},
  year = {2022},
  doi = {10.1016/j.cviu.2021.103352},
}
Figure