VIPS: Learning-View-Invariant Feature for Person Search

Hexu Wang, Wenlong Luo, Wei Wu, Fei Xie, Jindong Liu, Jing Li, Shizhou ZhangSensors 2025 · DOI

TL;DR

This paper introduces VIPS, a novel framework for person search in challenging UAV surveillance scenarios. VIPS tackles the significant issue of viewpoint variations by combining a Faster R-CNN detector with a view-invariant Re-Identification (VIReID) module that leverages large vision-language models (VLMs) for robust cross-camera matching.

What Problem Does This Paper Address?

Unmanned Aerial Vehicles (UAVs) are increasingly used for surveillance, capturing vast amounts of visual data. A key task is "cross-platform person search," which involves identifying a specific individual across different cameras, including those mounted on UAVs and ground-based systems (like CCTV). The major hurdle here is the extreme variation in viewpoints. UAVs capture images from high altitudes, leading to drastic geometric distortions, scale differences, and appearance shifts compared to ground-level cameras. These variations severely degrade the performance of traditional person search models, causing high rates of false negatives. Existing methods often struggle to maintain robustness when matching individuals across these disparate camera perspectives.

Methods

The proposed VIPS framework is a two-stage approach:

  1. Faster R-CNN for Detection: A standard Faster R-CNN model is used to detect bounding boxes of persons in the input images. These detected bounding boxes are then cropped and resized for the next stage.
  2. View-Invariant Re-Identification (VIReID) Module: This is the core innovation. It builds upon the CLIP-ReID model, which uses the vision-language model CLIP to align image and text features. VIPS enhances CLIP-ReID with two novel components:
  3. Mask Generator: To combat noise from occlusions and cluttered backgrounds in UAV images, a learnable mask generator is introduced. This module creates a foreground mask for the person, effectively purifying the image features by filtering out irrelevant background information. This helps the text encoder learn more accurate semantic features.
  4. View Prompts: To explicitly address viewpoint variations, VIPS designs "view prompts." These are learnable embeddings that are fed into the image encoder along with image patch embeddings. By encoding camera perspective information, these prompts help the model learn view-invariant features, making it more robust to different camera angles.

The VIReID module uses a two-stage training strategy: first, learning ID descriptor text features, and second, learning visual features guided by these text features.

Datasets

The paper evaluates VIPS on several datasets:

  • Person Search Datasets:
    • CUHK-SYSU: A large-scale dataset with realistic search scenarios.
  • PRW: Features pedestrians in real-world scenarios with synchronized cameras.
  • G2APS: A newly constructed dataset specifically designed for UAV-ground camera cross-platform person search, featuring significant viewpoint differences.
  • Person Re-Identification (ReID) Datasets (to evaluate VIReID module):
    • Market1501
    • MSMT17
    • Occluded-Duke

Key Insights and Ideas

  • VLMs as a Bridge for Cross-View Matching: The paper highlights the power of large vision-language models (like CLIP) in bridging the semantic gap between images from different viewpoints. By aligning text-based descriptors with visual features, VIPS can achieve robust matching even when camera perspectives are drastically different.
  • Targeted Noise Reduction for Text Features: The mask generator is a clever way to improve the quality of text features used for person identification. By removing background clutter and occlusions, it ensures that the text encoder focuses on the person's actual semantic information, leading to more accurate representations.
  • Explicitly Encoding View Information: Instead of implicitly learning to handle viewpoint changes, VIPS explicitly encodes this information using "view prompts." This direct approach allows the model to better adapt to and overcome the challenges posed by diverse camera angles, especially in UAV scenarios.
  • Viewpoint Variation is a Fundamental Challenge: The results strongly suggest that viewpoint variation is a primary bottleneck in person search, particularly in complex surveillance settings. VIPS's success demonstrates that addressing this specific challenge is crucial for advancing the field.

Key Results

  • State-of-the-Art Performance: VIPS achieves state-of-the-art results on the evaluated person search benchmarks, outperforming existing methods like HKD and TCTS. For instance, VIPS achieves significant mAP gains of +2.8% and +2.9% over HKD on PRW and CUHK-SYSU, respectively, and +9.5% and +4.3% over TCTS on PRW and CUHK-SYSU.
  • Strong VIReID Performance: The VIReID module also demonstrates competitive performance on person ReID datasets, achieving remarkable mAP scores (e.g., 90.3% on Market1501, 74.9% on MSMT17, and 60.2% on Occluded-Duke), surpassing existing CNN and ViT-based approaches.
  • Ablation Study Validation: Ablation studies confirm the effectiveness of both the mask generator and view prompt modules. Integrating both components leads to the best performance, with gains of +0.8% mAP and +1.6% top-1 accuracy over the baseline on PRW.
  • Visualizations: t-SNE visualizations show that the view prompt effectively brings feature distributions from different views closer together, demonstrating its ability to learn view-invariant features. Mask visualizations show the generator effectively suppresses background and occlusions.

Limitations and Future Work

  • Dataset Specificity: While VIPS shows strong performance, the paper notes that performance gains vary across datasets. Datasets with more extreme viewpoint differences (like G2APS) show larger improvements, suggesting that the method is particularly well-suited for such challenging scenarios.
  • Hyperparameter Sensitivity: The paper explores the sensitivity to view prompt parameters (depth h and token count N_v), finding optimal values (h=6, N_v=11) that were used for subsequent experiments. This indicates that careful tuning of these new components is necessary.
  • Future Work: The authors suggest that their work highlights the potential of VLM-derived semantic alignment for UAV applications, offering insights for future research in real-time UAV-based surveillance systems. The explicit encoding of camera perspectives could be further explored for more complex multi-camera coordination tasks.

Summary updated

Abstract

Unmanned aerial vehicles (UAVs) have become indispensable tools for surveillance, enabled by their ability to capture multi-perspective imagery in dynamic environments. Among critical UAV-based tasks, cross-platform person search-detecting and identifying individuals across distributed camera networks-presents unique challenges. Severe viewpoint variations, occlusions, and cluttered backgrounds in UAV-captured data degrade the performance of conventional discriminative models, which struggle to maintain robustness under such geometric and semantic disparities. To address this, we propose <b>v</b>iew-<b>i</b>nvariant <b>p</b>erson <b>s</b>earch (VIPS), a novel two-stage framework combining Faster R-CNN with a view-invariant re-Identification (VIReID) module. Unlike conventional discriminative models, VIPS leverages the semantic flexibility of large vision-language models (VLMs) and adopts a two-stage training strategy to decouple and align text-based ID descriptors and visual features, enabling robust cross-view matching through shared semantic embeddings. To mitigate noise from occlusions and cluttered UAV-captured backgrounds, we introduce a learnable mask generator for feature purification. Furthermore, drawing from vision-language models, we design view prompts to explicitly encode perspective shifts into feature representations, enhancing adaptability to UAV-induced viewpoint changes. Extensive experiments on benchmark datasets demonstrate state-of-the-art performance, with ablation studies validating the efficacy of each component. Beyond technical advancements, this work highlights the potential of VLM-derived semantic alignment for UAV applications, offering insights for future research in real-time UAV-based surveillance systems.
BibTeX
@inproceedings{wang2025vips,
  author = {Hexu Wang and Wenlong Luo and Wei Wu and Fei Xie and Jindong Liu and Jing Li and Shizhou Zhang},
  title = {{VIPS: Learning-View-Invariant Feature for Person Search}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25175362},
}
Figure