Introduction
Traditional person re-identification (ReID) typically operates on cropped images, simplifying the task to a direct visual matching problem. However, real-world surveillance scenarios involve complex "full-scene" environments characterized by multiple individuals, significant occlusions, and varying lighting conditions. To address these challenges, we introduce PS-ReID (Person-aware Segmentation and Re-identification), a framework that unifies person search and pixel-level segmentation using multimodal intelligence.
The PS-ReID Framework
PS-ReID leverages a Large Multimodal Model (LMM) backbone to treat ReID as a reasoning task. Unlike conventional methods, it integrates both visual and textual queries to disambiguate targets in cluttered scenes. This approach allows the model to "understand" descriptive attributes—such as clothing patterns or specific accessories—and locate the corresponding individual within a holistic scene.
Methodology
The architecture is defined by several key innovations:
- LMM Backbone: Utilizing LLaVA (7B or 13B), the model performs high-level reasoning to identify individuals based on complex descriptions.
- Dual-Path Asymmetric Encoding: To prevent semantic entanglement between the query and the target scene, the model employs independent adapters. A Region-to-Text adapter extracts identity cues from the query, while an Image-to-Text adapter reasons over the holistic target scene.
- Token-Level ReID Loss: A novel loss function operates on internal LMM tokens (
[QRY],[SEG],[REJ]), aligning the model's internal reasoning space with the retrieval objective. - Segmentation Module: Upon identifying a match, a segmentation head—inspired by the Segment Anything Model (SAM)—generates a precise binary mask of the target.
The $M^2$ReID Dataset
To support this multimodal task, we developed $M^2$ReID, the largest full-scene multimodal ReID dataset to date. It contains over 204,000 images and 4,894 unique identities. Each entry includes high-quality segmentation masks and descriptive text queries generated via GPT-4o and human refinement, capturing diverse indoor and outdoor environments across varying light levels.
Experimental Results
PS-ReID demonstrates significant improvements over existing state-of-the-art models such as LISA and SEEM. On the challenging TestG500 gallery, the PS-ReID-13B variant achieved:
- 72.1% Top-1 Accuracy and 60.0% mAP, nearly tripling the performance of fine-tuned baselines.
- 39.8% gIoU, showcasing superior segmentation precision in crowded environments.
Qualitative analysis confirms the model's robustness under severe occlusion and low-light conditions, where unimodal approaches often fail to maintain identity consistency.
Conclusion
By treating person re-identification as a multimodal reasoning and segmentation task, PS-ReID moves the field closer to practical, real-world application. The integration of text and image queries provides a synergistic effect that overcomes the limitations of traditional cropped-image matching. Future work will focus on enhancing semantic disambiguation for individuals with highly similar appearances and improving performance under extreme occlusion.
Summary updated