TL;DR
SA-Person introduces a scene-aware paradigm for text-based person retrieval, moving beyond pure appearance matching by integrating global environmental context. By combining a fast appearance-based filter with a training-free Multimodal Large Language Model (MLLM) re-ranker, it achieves state-of-the-art accuracy while maintaining computational scalability.
What Problem Does This Paper Address?
Traditional Text-Based Person Retrieval (TBPR) focuses almost exclusively on matching appearance descriptions (e.g., "a girl wearing a green shirt") to cropped pedestrian images. However, in real-world surveillance or crowded scenes, multiple people often wear similar clothes, leading to high ambiguity.
Humans naturally overcome this by describing the surrounding context (e.g., "sitting near a window in a restaurant"). Existing models struggle to leverage these environmental cues because they operate primarily on isolated, cropped image patches, and full-scene images often introduce distracting background clutter that overwhelms standard appearance-based embeddings.
Methods
The authors propose SA-Person, a two-stage region-to-global cascade retrieval framework:
- Stage 1: Appearance-based Retrieval: An off-the-shelf detector identifies and crops pedestrians from full-scene images. A fine-tuned dual-encoder model (initialized with CLIP) matches appearance-only text queries against these cropped regions. This step efficiently filters a massive gallery down to a compact set of top-$K$ candidates.
- Stage 2: Scene-aware Re-ranking (SceneRanker): This training-free, plug-and-play module leverages a Multimodal Large Language Model (MLLM, such as InternVL). It takes the full-scene images corresponding to the top-$K$ candidates, visually prompts them using bounding boxes, and uses a structured prompt to re-rank the candidates based on the full context-enriched text query.
By confining expensive MLLM inference to just the top-$K$ candidates, the framework avoids the prohibitive computational cost of running a large language model across an entire gallery.
Datasets
The paper evaluates its approach on four benchmarks:
- SCENEPERSON-13W: A newly introduced large-scale dataset containing 131,438 pedestrian instances across 112,872 real-world scenes extracted from YouTube videos. It provides rich descriptions combining fine-grained pedestrian attributes and global scene context (landmarks, proximity, relations).
- Augmented CUHK-SYSU: Modified with MLLM-generated context-enriched descriptions for preliminary analysis and surveillance evaluation.
- Urban1K & ShareGPT4V (5K subset): Used to test open-domain generalization.
Key Insights and Ideas
- The Complementary Strengths Paradox: Domain-specific TBPR models excel at localized appearance matching but choke on full-scene background noise. Conversely, zero-shot MLLMs possess incredible holistic scene reasoning but lack fine-grained domain alignment for gallery-wide searches. Cascading them yields the best of both worlds.
- Prompting Strategy Matters: Textually embedding bounding box coordinates into the MLLM prompt (Box Embedded Prompt) yields better results than visually overlaying a red box on the image. Direct visual highlighting can introduce artifacts that disrupt the MLLM's holistic scene understanding.
- Scalability via Cascading: Shifting from a gallery-wide MLLM traversal $O(N)$ to a cascade approach $O(N\mu_s + K\mu_m)$ makes scene-aware retrieval computationally viable for large databases.
Key Results
- On the new SCENEPERSON-13W dataset, SA-Person achieves a 78.34% Rank@1 and 82.86% mAP (using a ViT-L/14 backbone), outperforming the strongest baseline (IRRA) by 14.97% in Rank@1.
- Integrating SceneRanker as a plug-and-play module consistently boosts various baseline models (like TBPS-CLIP, RaSa, MARS) by over 14% in Rank@1.
- In open-domain contexts (Urban1K), SceneRanker enhances baseline CLIP performance by over 19% in Rank@1 and 12% in mAP, proving its capability to exploit urban contextual cues effectively.
Limitations and Future Work
- Stylistic Uniformity: The textual descriptions in SCENEPERSON-13W were generated using an MLLM, which can lead to a relatively uniform stylistic phrasing compared to the diverse variations found in real-world human text.
- Off-the-shelf Constraints: SceneRanker relies entirely on off-the-shelf MLLMs without task-specific fine-tuning. The authors note that exploring lightweight adapters or domain-adaptive pretraining could further amplify precision in retrieval intent alignment without sacrificing zero-shot robustness.