UPAR: Unified Pedestrian Attribute Recognition and Person Retrieval

Andreas Specker, Mickael Cormier, Jürgen BeyererWACV 2023

TL;DR

This paper introduces UPAR, a novel, large-scale dataset that unifies four existing pedestrian attribute recognition (PAR) datasets to enable robust generalization evaluation. They also propose a strong baseline for PAR and attribute-based person retrieval, demonstrating state-of-the-art performance and highlighting the importance of specific regularization techniques for cross-domain generalization.

What Problem Does This Paper Address?

Recognizing soft-biometric attributes of pedestrians (e.g., clothing color, presence of a backpack) is crucial for applications like video surveillance and fashion retrieval. While deep learning models have shown promise, their ability to generalize across different datasets, viewpoints, lighting conditions, and resolutions remains poorly understood. This is largely due to existing datasets being biased towards specific scenarios and having limited overlap in annotated attributes, making cross-dataset evaluation and true generalization assessment difficult. The paper aims to bridge this gap by creating a unified dataset and a benchmark that facilitates systematic investigation into generalizable PAR.

Methods

The core of the paper's methodology lies in two main contributions:

  • UPAR Dataset Construction: The authors unified four popular PAR datasets (PA100K, PETA, RAPv2, and Market1501) by adding over 3.3 million new annotations. This harmonization effort resulted in 40 binary attributes across 12 categories, covering a wide range of visual characteristics. This unified dataset allows for cross-domain evaluation, a critical aspect missing in previous work.
  • Strong Baseline for PAR and Retrieval: To address the challenges of generalization, the authors developed a robust baseline. This involved a thorough analysis of design choices, including:
  • Backbone Architectures: Comparing ResNet50 with newer models like Swin Transformer and ConvNeXt-B, finding ConvNeXt-B to be superior.
  • Regularization Techniques: Investigating Exponential Moving Average (EMA), optimal batch size, Dropout, label smoothing, and AdamW optimizer. They found that Dropout on the last feature layer and AdamW significantly improved performance, especially for generalization.
  • Data Augmentation: Evaluating Random Erasing (RE) and AugMix (AM), with AugMix showing benefits for generalization.

Datasets

The paper introduces the UPAR dataset, which is a unification of:

  • PA100K: Large-scale outdoor dataset.
  • PETA: Dataset with diverse scenarios.
  • RAPv2: Dataset focusing on indoor scenes.
  • Market1501: Dataset with outdoor campus imagery.

The UPAR dataset contains 224,737 images with annotations for 40 binary attributes over 12 categories. It is split into 148,048 training, 30,830 validation, and 45,859 test images, following the original dataset splits.

Key Insights and Ideas

  • Generalization is Key: The paper strongly emphasizes that current PAR models, while performing well on single datasets, often fail to generalize to unseen domains. UPAR is designed to explicitly address this limitation.
  • Unified Datasets Enable Real-World Evaluation: By harmonizing attributes across diverse datasets, UPAR provides a more realistic benchmark for evaluating models that need to perform across various conditions, mimicking real-world surveillance scenarios.
  • ConvNeXt-B Outperforms Older Backbones: For both PAR and retrieval tasks, the ConvNeXt-B architecture demonstrated superior performance compared to ResNet50 and Swin Transformers, suggesting its suitability for complex attribute recognition.
  • Specific Regularization is Crucial for Generalization: Techniques like Dropout on the final feature layer and the AdamW optimizer were found to be critical for improving generalization performance, more so than other methods like EMA which can be sensitive to batch size and data availability.
  • Attribute-based Retrieval Benefits from Diverse Data: The study suggests that attribute-based person retrieval tasks see significant improvements with more diverse training data, as provided by the UPAR dataset.

Key Results

  • The proposed UPAR dataset and baseline approach achieve state-of-the-art performance on both PAR and attribute-based person retrieval tasks across multiple UPAR splits and individual datasets.
  • The ConvNeXt-B backbone, combined with Dropout and AdamW, significantly improved generalization performance in the UPAR cross-validation (CV) and leave-one-out (LOO) settings. For instance, in the CV setting, the proposed method with ConvNeXt-B achieved an mAP of 17.2 ± 3.7 and R-1 of 20.1 ± 9.4, outperforming previous baselines.
  • The paper demonstrates that even with a simple baseline architecture, significant improvements can be achieved by carefully selecting regularization techniques and leveraging a comprehensive dataset like UPAR.

Limitations and Future Work

  • Dataset Biases Persist: While UPAR mitigates some biases by unifying datasets, it acknowledges that significant distribution shifts still exist concerning attribute distributions and image characteristics, influenced by the original data collection contexts (e.g., market, mall, campus). Future work should focus on representing more diverse communities and clothing styles.
  • Privacy Concerns: The paper touches upon the ethical dilemma of using real-world surveillance data and suggests that leveraging existing, well-accepted datasets is a viable strategy to avoid acquiring more personal data. However, the potential for malicious use of PAR systems (e.g., identifying individuals based on sensitive attributes) is a societal concern that needs careful consideration.
  • Generalization Gap: Despite achieving state-of-the-art results, the paper notes that a R-1 score of 15.7% in some generalization scenarios might not be sufficient for real-world applications, indicating room for further research in improving cross-domain retrieval.
  • Attribute Confusion: The paper raises a point about potential confusion between clothing color and skin color in low-resolution images, suggesting that further investigation into how PAR models handle such ambiguities is needed.

Summary updated

BibTeX
@inproceedings{specker2023upar,
  author = {Andreas Specker and Mickael Cormier and Jürgen Beyerer},
  title = {{UPAR: Unified Pedestrian Attribute Recognition and Person Retrieval}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}

Loading related content...

Figure