HorNet: A Hierarchical Offshoot Recurrent Network for Improving Person Re-ID via Image Captioning

Shiyang Yan, Jun Xu, Yuai Liu, Lin Xu

Abstract

Person re-identification (re-ID) is a critical task in surveillance, involving the matching of individuals across non-overlapping camera views. Traditional methods often struggle with visual variations caused by changes in lighting, pose, and viewpoint. This paper introduces HorNet, a Hierarchical Offshoot Recurrent Network that leverages the semantic stability of language to augment visual features. By utilizing image captions—either existing or synthetically generated through a domain-transfer pipeline—HorNet employs discrete gates to filter out noise and redundant information. Experimental results on Market-1501, CUHK03, and Duke-MTMC datasets demonstrate that integrating refined linguistic descriptions significantly improves re-ID accuracy.

Introduction: The Stability of Language in Re-ID

Matching the same individual across different cameras is notoriously difficult because a person’s visual appearance (pixels) can change drastically. However, the semantic description of a person—such as "wearing a light blue collared shirt"—remains constant regardless of the camera angle or lighting conditions.

Overview of the HorNet framework for person re-identification using image captions.

Despite the potential of language, two major hurdles exist:

  1. Data Scarcity: Most re-ID datasets lack textual descriptions.
  2. Noisy Text: Automatically generated captions often contain errors or irrelevant details that can degrade model performance.

HorNet addresses these by combining a robust caption generation pipeline with a novel hierarchical architecture designed to extract only the most relevant semantic features.

Methodology

The proposed framework operates in two primary stages: caption preparation and feature fusion via the HorNet architecture.

Caption Generation & Domain Adaptation

For datasets without human-annotated captions, the authors employ a Similarity Preserving Generative Adversarial Network (SPGAN). This network transfers the visual style of unlabelled target datasets (e.g., Duke-MTMC) to match a source dataset that already contains captions (e.g., CUHK-PEDES). Once the domain gap is bridged, a pre-trained image captioner generates descriptions for every image in the target dataset.

The HorNet Architecture

The core of the system is the Hierarchical Offshoot Recurrent Network (HorNet), which processes visual and textual data in parallel.

Detailed architecture of HorNet, highlighting the hierarchical LSTM with discrete Gumbel-sigmoid gates.
  • Visual Branch: Uses a standard ResNet-50 backbone to extract high-level visual features.
  • Language Branch: Features a two-layer LSTM. The bottom layer processes the raw word sequence, while the upper layer incorporates discrete binary gates (implemented via the Gumbel-sigmoid trick). These gates act as a filter, deciding which words are visually grounded and useful for re-ID, effectively ignoring redundant or incorrect generated text.
  • Joint Representation: The filtered language features are concatenated with visual features. The entire model is trained end-to-end using a combination of Identification (ID) Loss and Triplet Loss.

Experimental Results

HorNet was evaluated on three major benchmarks, showing substantial improvements over visual-only baselines:

Dataset Metric Baseline (Visual Only) HorNet (Visual + Text)
CUHK03 (Detected) Top-1 Acc 72.4% 95.0%
Market-1501 mAP - 85.6%
Duke-MTMC mAP - 79.2%

Notably, on Duke-MTMC, where captions were entirely synthetic and generated through the domain-transfer pipeline, the model still achieved high performance. This proves that even imperfect AI-generated captions provide valuable regularizing information when filtered correctly.

Key Insights

  • Semantic Regularization: Language provides a "canonical" description that is invariant to physical environment changes, acting as a powerful anchor for visual features.
  • The Necessity of Filtering: Direct use of generated text is often counterproductive due to noise. The discrete gates in HorNet are essential for selecting reliable semantic cues.
  • Bridging the Domain Gap: Using SPGAN to align image styles before captioning significantly improves the quality and relevance of the generated descriptions.

Conclusion

HorNet demonstrates that linguistic information is a potent tool for overcoming the inherent limitations of visual-only person re-identification. By intelligently filtering noisy captions through a hierarchical recurrent structure, the model achieves state-of-the-art results. Future work will focus on developing captioning modules specifically optimized for re-ID tasks and exploring more sophisticated fusion strategies for multi-modal features.

Summary updated

BibTeX
@inproceedings{yan2019hornet,
  author = {Shiyang Yan and Jun Xu and Yuai Liu and Lin Xu},
  title = {{HorNet: A Hierarchical Offshoot Recurrent Network for Improving Person Re-ID via Image Captioning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/742},
}
Figure