Parameter sharing and multi-granularity feature learning for cross-modality person re-identification

Sixian Chan, Feng Du, Tinglong Tang, Guodao Zhang, Xiaoliang Jiang, Qiu GuanComplex 2023 · DOI

TL;DR

This paper introduces PSFLNet, a framework designed to match pedestrian images across visible and infrared spectrums. By strategically sharing convolutional parameters and combining global body structure with fine-grained local details, the authors achieve state-of-the-art performance on major cross-modality benchmarks.

What Problem Does This Paper Address?

Person Re-identification (Re-ID) is essentially an image retrieval task: "Given this image of a person from Camera A, find them in Camera B." While standard Re-ID (visible-to-visible) is well-studied, Visible-Infrared Re-ID (VI-ReID) is significantly harder.

The authors identify two primary hurdles:

  1. The Modality Gap: Visible (RGB) images have three channels and rich color/texture, while Infrared (IR) images are single-channel and capture thermal information.
  2. Feature Trade-offs: Existing methods often focus on "part-level" features (slicing an image into horizontal strips) to ignore background noise. However, this often destroys the global semantic structure—like the overall body contour—which is vital for matching across different lighting conditions.

Methods

The authors propose the Parameter Sharing and Feature Learning Network (PSFLNet), which consists of three main innovations:

  • Two-Stream Parameter Sharing Network (TPSN): Instead of just sharing the final fully connected layers (which only handle 1D vectors), PSFLNet splits a ResNet50 backbone. The early layers are modality-specific to handle unique RGB/IR traits, while the later convolutional layers are shared. This allows the network to learn modality-invariant features while preserving 3D spatial structures.
  • Multi-Granularity Feature Learning (MGFL): To avoid the pitfalls of only looking at "parts," this module jointly learns global features (the whole body) and local features (horizontal strips). This ensures the model doesn't lose the "big picture" while still paying attention to specific details like shoes or bags.
  • Center Alignment Loss ($L_{ca}$): A novel loss function designed for global features. It encourages the network to cluster features of the same identity toward a central point in the feature space, making the representations more discriminative and reducing the distance between visible and infrared versions of the same person.

The pipeline also utilizes a Middle Modality Generator (MMG), which creates "fake" intermediate images to help bridge the visual gap during training.

Datasets

The model was evaluated on two cornerstone datasets for cross-modality Re-ID:

  1. SYSU-MM01: A large-scale dataset containing images from four visible and two infrared cameras, featuring both indoor and outdoor environments.
  2. RegDB: Collected using dual-camera systems (one visible, one thermal) where images are perfectly aligned, containing 412 identities.

Key Insights and Ideas

  • Conv-Sharing vs. FC-Sharing: Most previous work shared parameters only at the very end of the network (Fully Connected layers). The authors argue that sharing convolutional layers is superior because it preserves the spatial relationship of body parts, which is a powerful cue for identification regardless of the light spectrum.
  • The "Middle Modality" bridge: By generating images that sit halfway between visible and infrared, the network has a "stepping stone" to learn features that work for both, rather than trying to jump the massive gap in one go.
  • Global + Local is Key: The results confirm that while horizontal slicing (local features) helps mitigate background interference, the global contour (global features) provides the necessary context to make those local features meaningful.

Key Results

PSFLNet outperformed several state-of-the-art (SOTA) methods, including both feature-extraction and GAN-based approaches.

  • SYSU-MM01 (All-Search Mode): Achieved a Rank-1 accuracy of 74.0% and a mean Average Precision (mAP) of 70.51%. This represents a significant jump (+3.4% Rank-1) over their baseline.
  • RegDB (Visible-to-Infrared): Reached an impressive 95.87% Rank-1 accuracy and 91.08% mAP.
  • Ablation Studies: The authors showed that sharing layers 2 through 4 of the ResNet backbone provided the optimal balance between learning modality-specific details and modality-invariant structures.

Limitations and Future Work

The authors acknowledge that their method still relies on horizontal slicing for part-level features. This is a "blind" method that assumes pedestrians are always centered and upright. If a person is partially occluded or in a strange pose, the slices might not align correctly.

Future Direction: The team plans to investigate image segmentation to replace horizontal slicing. By explicitly segmenting the human body from the background, the model could extract foreground-only features more accurately, further reducing the "noise" from complex backgrounds.

Summary updated

Abstract

Abstract Visible-infrared person re-identification aims to match pedestrian images between visible and infrared modalities, and its two main challenges are intra-modality differences and cross-modality differences between visible and infrared images. To address these issues, many advanced methods attempt to design new network structures to extract modality-sharing features, mitigate modality differences, or learn part-level features to overcome background interference. However, they ignore the parameter sharing of the convolutional layers to obtain more modality-sharing features. At the same time, only using part-level features lack discriminative pedestrian representations such as body structure and contours. To handle these problems, a parameter sharing and feature learning network is proposed in this paper to mitigate modality differences and further enhance feature discrimination. Firstly, a new two-stream parameter sharing network is proposed, by sharing the convolutional layers parameters to obtain more modality-sharing features. Secondly, a multi-granularity feature learning module is designed to reduce modality differences at both coarse and fine-grained levels while further enhancing feature discriminability. In addition, a center alignment loss is proposed to learn relationships between identities and to reduce modality differences by clustering features into their centers. For the part-level feature learning, the hetero-center triplet loss is adopted to alleviate the strict constraints of triplet loss. Finally, extensive experiments are conducted to validate our method outperforms state-of-the-art methods on two challenging datasets. In the SYSU-MM01 dataset, the Rank-1 and mAP reach $$74.0\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>74.0</mml:mn> <mml:mo>%</mml:mo> </mml:mrow> </mml:math> and $$70.51\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>70.51</mml:mn> <mml:mo>%</mml:mo> </mml:mrow> </mml:math> in the all-search mode, which is an increase of $$3.4\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>3.4</mml:mn> <mml:mo>%</mml:mo> </mml:mrow> </mml:math> and $$3.61\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>3.61</mml:mn> <mml:mo>%</mml:mo> </mml:mrow> </mml:math> to baseline, respectively.
BibTeX
@inproceedings{chan2023parameter,
  author = {Sixian Chan and Feng Du and Tinglong Tang and Guodao Zhang and Xiaoliang Jiang and Qiu Guan},
  title = {{Parameter sharing and multi-granularity feature learning for cross-modality person re-identification}},
  booktitle = {Complex},
  year = {2023},
  doi = {10.1007/s40747-023-01189-y},
}
Figure