HorNet: A Hierarchical Offshoot Recurrent Network for Improving Person Re-ID via Image Captioning

Shiyang Yan, Jun Xu, Yuai Liu, Lin Xu

Introduction

Person Re-identification (re-ID) is a critical task in surveillance and computer vision, aimed at matching individuals across non-overlapping camera views. The primary challenge lies in significant appearance variance caused by changes in lighting, viewpoint, and human pose. While traditional methods focus on visual feature extraction, they often struggle with these inconsistencies. This paper introduces HorNet, a framework that leverages natural language descriptions to provide a stable semantic anchor, enhancing the robustness of person re-ID systems.

Overview of the HorNet framework, illustrating the use of both available and generated captions for person re-identification.

Methodology

The HorNet architecture is designed to fuse visual information with textual descriptions through a hierarchical recurrent structure.

1. Caption Generation and Domain Transfer

For datasets lacking textual annotations, the authors employ a Similarity Preserving Generative Adversarial Network (SPGAN) to perform domain transfer. This aligns the style of the target dataset images with the training data of a pre-trained image captioner, ensuring the generation of high-quality, relevant descriptions.

2. Hierarchical Offshoot Recurrent Network (HorNet)

The core of the system is a two-branch network:

  • Visual Branch: Utilizes a ResNet-50 backbone to extract global visual features.
  • Language Branch: Employs a two-layer LSTM. To address the noise and redundancy inherent in natural language, the model introduces discrete binary gates between the LSTM layers. These gates are controlled by visual features via a Gumbel-sigmoid function, allowing the network to selectively focus on words that are visually grounded.
Detailed architecture of HorNet, highlighting the ResNet50 visual branch and the hierarchical LSTM with Gumbel-Sigmoid gating mechanisms.

3. Joint Optimization

The model is trained using a combination of Identification (ID) Loss and Triplet Loss. This joint learning strategy ensures that the fused visual-language representation is highly discriminative for person matching.

Experiments and Results

The researchers evaluated HorNet on three major benchmarks: CUHK03, Market-1501, and Duke-MTMC-reID.

  • CUHK03: HorNet achieved a state-of-the-art Rank-1 accuracy of 97.1% on labeled data and 95.0% on detected data, significantly outperforming previous vision-and-language approaches.
  • Market-1501: The model reached 91.0% Rank-1 accuracy and 85.8% mAP when integrated with re-ranking techniques.
  • Duke-MTMC-reID: Even using synthetic captions generated via domain transfer, the model achieved 79.2% mAP, demonstrating the effectiveness of the proposed captioning pipeline.

Conclusion

HorNet demonstrates that natural language descriptions can serve as powerful, viewpoint-invariant features for person re-identification. By introducing a hierarchical gating mechanism, the model effectively filters textual noise, focusing on the most relevant semantic information. Future work will focus on improving the precision of re-ID specific captioning and scaling the framework to larger, more diverse datasets.

Summary updated

BibTeX
@inproceedings{yan2019hornet,
  author = {Shiyang Yan and Jun Xu and Yuai Liu and Lin Xu},
  title = {{HorNet: A Hierarchical Offshoot Recurrent Network for Improving Person Re-ID via Image Captioning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/742},
}
Figure