Taking Modality-free Human Identification as Zero-shot Learning

Zhizhe Liu, Xingxing Zhang, Zhenfeng Zhu, Shuai Zheng, Yao Zhao, Jian Cheng

Introduction

Traditional human identification systems, such as face recognition or person re-identification (re-ID), primarily operate in an Image-to-Image (I2I) paradigm. In this setting, a query image is matched against a gallery of known images. However, real-world scenarios often involve "modality gaps" where queries or galleries may consist of textual attribute descriptions rather than raw images. Examples include Attribute-to-Image (A2I) retrieval based on witness descriptions or Image-to-Attribute (I2A) matching for privacy-preserving identification.

Overview of human identification tasks across different modalities: Image-to-Image (I2I), Image-to-Attribute (I2A), and Attribute-to-Image (A2I).

These cross-modal tasks are inherently Zero-Shot Learning (ZSL) problems because the specific individuals being searched for are often not present in the training data. Standard ZSL methods struggle with human identification due to low inter-class similarity (humans look alike) and high intra-class variation (lighting, pose, and camera changes).

The MFHI Framework

To address these challenges, this paper introduces the Modality-Free Human Identification (MFHI) framework. MFHI maps both visual images and textual attribute descriptions into a shared embedding space, allowing for seamless identification regardless of the input format.

The framework is built upon three core components:

  1. Semantics-Guided Spatial Attention (SGSA): This module moves beyond global image features by focusing on attribute-specific regions. It utilizes an Attribute Prediction Module (APM) and an Attribute Attention Module (AAM) to generate heatmaps. For instance, if the model identifies "backpack" as a relevant attribute, the attention mechanism prioritizes the person's back.
  2. Prototype Learning Module (PLM): PLM converts category-level textual descriptions into "identity prototypes" within the shared mathematical space. This enables direct comparison between visual feature vectors and text-derived vectors.
  3. Distribution Consistency Constraint (DCM): To ensure precise alignment between modalities, the authors employ an angular margin loss (inspired by ArcFace). This constraint forces features of the same identity to cluster tightly on a hypersphere, maximizing the separation between distinct individuals.
The MFHI framework architecture, featuring Semantics-Guided Spatial Attention (SGSA) and Identity Prototype Learning.

Evaluation and Results

The MFHI framework was evaluated on four major benchmarks: CelebA and LFWA for face identification, and Market-1501 and DukeMTMC-reID for person re-identification.

MFHI consistently outperformed state-of-the-art (SOTA) methods across all tasks:

  • Face Identification: On the CelebA dataset (I2A), MFHI achieved a 30.1% Top-1 accuracy, a significant improvement over the previous best ZSL method, AREN (24.3%).
  • Person Re-ID: On Market-1501 (A2I), MFHI reached 44.8% Rank@1 accuracy, surpassing specialized models like AIHM (43.3%) and standard ZSL approaches like DEM (34.0%).

Qualitative analysis confirmed that the model effectively localizes relevant body parts—such as focusing on legs for "pants" or the head for "hats"—when performing cross-modal matching.

Conclusion

By framing human identification as a zero-shot learning problem, MFHI provides a robust, modality-free solution for real-world surveillance and security. The use of attributes as spatial anchors significantly enhances the discriminative power of visual features. Future work will explore adaptive prototype learning to better represent the complex appearance distributions of individuals across different environments and timeframes.

Summary updated

Abstract

Human identification is an important topic in event detection, person tracking, and public security. There have been numerous methods proposed for human identification, such as face identification, person re-identification, and gait identification. Typically, existing methods predominantly classify a queried image to a specific identity in an image gallery set ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">I2I</i> ). This is seriously limited for the scenario where only a textual description of the query or an attribute gallery set is available in a wide range of video surveillance applications ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A2I</i> or <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">I2A</i> ). However, very few efforts have been devoted towards modality-free identification, i.e., identifying a query in a gallery set in a scalable way. In this work, we take an initial attempt, and formulate such a novel <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</b> odality- <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</b> ree <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</b> uman <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">I</b> dentification (named MFHI) task as a generic zero-shot learning model in a scalable way. Meanwhile, it is capable of bridging the visual and semantic modalities by learning a discriminative prototype of each identity. In addition, the semantics-guided spatial attention is enforced on visual modality to obtain interpretable representations with both high global category-level and local attribute-level discrimination. Finally, we design and conduct an extensive group of experiments on two common challenging identification tasks, including face identification and person re-identification, demonstrating that our method outperforms a wide variety of state-of-the-art methods on modality-free human identification.
BibTeX
@article{liu2022taking,
  author = {Zhizhe Liu and Xingxing Zhang and Zhenfeng Zhu and Shuai Zheng and Yao Zhao and Jian Cheng},
  title = {{Taking Modality-free Human Identification as Zero-shot Learning}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2022},
  url = {https://openreview.net/forum?id=G5Sy0jlncW},
}
Figure