Introduction
Traditional human identification systems, such as face recognition or person re-identification (re-ID), primarily operate in an Image-to-Image (I2I) paradigm. In this setting, a query image is matched against a gallery of known images. However, real-world scenarios often involve "modality gaps" where queries or galleries may consist of textual attribute descriptions rather than raw images. Examples include Attribute-to-Image (A2I) retrieval based on witness descriptions or Image-to-Attribute (I2A) matching for privacy-preserving identification.
These cross-modal tasks are inherently Zero-Shot Learning (ZSL) problems because the specific individuals being searched for are often not present in the training data. Standard ZSL methods struggle with human identification due to low inter-class similarity (humans look alike) and high intra-class variation (lighting, pose, and camera changes).
The MFHI Framework
To address these challenges, this paper introduces the Modality-Free Human Identification (MFHI) framework. MFHI maps both visual images and textual attribute descriptions into a shared embedding space, allowing for seamless identification regardless of the input format.
The framework is built upon three core components:
- Semantics-Guided Spatial Attention (SGSA): This module moves beyond global image features by focusing on attribute-specific regions. It utilizes an Attribute Prediction Module (APM) and an Attribute Attention Module (AAM) to generate heatmaps. For instance, if the model identifies "backpack" as a relevant attribute, the attention mechanism prioritizes the person's back.
- Prototype Learning Module (PLM): PLM converts category-level textual descriptions into "identity prototypes" within the shared mathematical space. This enables direct comparison between visual feature vectors and text-derived vectors.
- Distribution Consistency Constraint (DCM): To ensure precise alignment between modalities, the authors employ an angular margin loss (inspired by ArcFace). This constraint forces features of the same identity to cluster tightly on a hypersphere, maximizing the separation between distinct individuals.
Evaluation and Results
The MFHI framework was evaluated on four major benchmarks: CelebA and LFWA for face identification, and Market-1501 and DukeMTMC-reID for person re-identification.
MFHI consistently outperformed state-of-the-art (SOTA) methods across all tasks:
- Face Identification: On the CelebA dataset (I2A), MFHI achieved a 30.1% Top-1 accuracy, a significant improvement over the previous best ZSL method, AREN (24.3%).
- Person Re-ID: On Market-1501 (A2I), MFHI reached 44.8% Rank@1 accuracy, surpassing specialized models like AIHM (43.3%) and standard ZSL approaches like DEM (34.0%).
Qualitative analysis confirmed that the model effectively localizes relevant body parts—such as focusing on legs for "pants" or the head for "hats"—when performing cross-modal matching.
Conclusion
By framing human identification as a zero-shot learning problem, MFHI provides a robust, modality-free solution for real-world surveillance and security. The use of attributes as spatial anchors significantly enhances the discriminative power of visual features. Future work will explore adaptive prototype learning to better represent the complex appearance distributions of individuals across different environments and timeframes.
Summary updated