Fairest of Them All: Establishing a Strong Baseline for Cross-Domain Person ReID

Devinder Kumar, Parthipan Siva, Paul Marchwica, Alexander WongarXiv 2019 · OpenReview

Abstract

Person re-identification (ReID) remains a very difficult challenge in computer vision, and critical for large-scale video surveillance scenarios where an individual could appear in different camera views at different times. There has been recent interest in tackling this challenge using cross-domain approaches, which leverages data from source domains that are different than the target domain. Such approaches are more practical for real-world widespread deployment given that they don't require on-site training (as with unsupervised or domain transfer approaches) or on-site manual annotation and training (as with supervised approaches). In this study, we take a systematic approach to establishing a large baseline source domain and target domain for cross-domain person ReID. We accomplish this by conducting a comprehensive analysis to study the similarities between source domains proposed in literature, and studying the effects of incrementally increasing the size of the source domain. This allows us to establish a balanced source domain and target domain split that promotes variety in both source and target domains. Furthermore, using lessons learned from the state-of-the-art supervised person re-identification methods, we establish a strong baseline method for cross-domain person ReID. Experiments show that a source domain composed of two of the largest person ReID domains (SYSU and MSMT) performs well across six commonly-used target domains. Furthermore, we show that, surprisingly, two of the recent commonly-used domains (PRID and GRID) have too few query images to provide meaningful insights. As such, based on our findings, we propose the following balanced baseline for cross-domain person ReID consisting of: i) a fixed multi-source domain consisting of SYSU, MSMT, Airport and 3DPeS, and ii) a multi-target domain consisting of Market-1501, DukeMTMC-reID, CUHK03, PRID, GRID and VIPeR.

Introduction: The Challenge of Cross-Domain ReID

Person re-identification (ReID) is the computer vision task of matching individuals across different camera views in a surveillance network. While supervised ReID models perform incredibly well, they require massive amounts of manually annotated data for every new camera network (the "target" domain), making real-world deployment expensive and impractical.

Cross-domain ReID aims to solve this by training a model on annotated "source" domains and deploying it zero-shot to unseen target domains. However, the field currently lacks standardized benchmarks. Researchers use wildly different combinations of source and target datasets for training and evaluation. This inconsistency makes it nearly impossible to tell if a newly proposed algorithm is actually better, or if it simply benefited from a more advantageous choice of training data. This paper tackles this gap by establishing a rigorous, standardized benchmark and a strong baseline model for cross-domain ReID.

Comparison of common person ReID paradigms, highlighting the differences between supervised, cross-domain, domain adaptation, and unsupervised approaches.

Methodology: Data Curation and Baseline Optimization

The authors focus on systematic data curation and applying proven "tricks of the trade" rather than proposing a radically new neural network architecture.

Domain Analysis

They systematically evaluate the similarity between existing datasets using cross-domain testing and UMAP (Uniform Manifold Approximation and Projection) embeddings. This helps identify "data leakage"—for example, testing on a dataset collected in the exact same environment as the training set.

Baseline Model

They build a strong baseline using a standard ResNet-50 backbone pre-trained on ImageNet. They incorporate best practices from recent supervised ReID research, including a combination of ID/softmax loss and triplet loss, a Batch Normalization Neck (BNNeck), and batch normalization at the end of the network. Interestingly, they found that data augmentation techniques like random erasing, which work well in supervised settings, actually hurt cross-domain performance and were specifically excluded.

Standardizing Benchmarks: Dataset Analysis

The study comprehensively analyzes 11 popular ReID datasets: Airport, DukeMTMC-reID, Market-1501, CUHK03, SYSU, MSMT17, CUHK02, PRID2011, GRID, VIPeR, and 3DPeS. Based on their analysis, they propose two standardized benchmark splits:

  • Fixed Split: A balanced multi-source domain (SYSU, MSMT, Airport, 3DPeS) evaluated against six distinct target domains (Market-1501, DukeMTMC-reID, CUHK03, PRID, GRID, VIPeR).
  • Leave-One-Out Split: A massive multi-source domain utilizing all available large datasets, leaving out only the specific target dataset being evaluated to maximize training diversity.

Key Insights and Results

  • Bigger is (usually) better: Incrementally adding datasets to the source domain generally improves generalization to unseen target domains. A robust source domain composed of multiple environments yields highly robust cross-domain results.
  • Beware of environmental overlap: UMAP analysis revealed that CUHK02 and CUHK03 are virtually identical environments. Using one as a source and the other as a target artificially inflates cross-domain performance.
  • Similarity beats diversity for specific targets: While a massive, diverse multi-source domain is best for general deployment, if a target domain happens to be highly similar to a specific source domain, adding more different datasets to the training mix can actually dilute the features and degrade performance slightly.
  • Small targets are too noisy: Datasets like GRID and PRID2011 have too few query images and identities to provide statistically meaningful insights for cross-domain evaluation.

Performance Benchmarks

The proposed multi-source baseline achieves highly competitive results without any target-domain adaptation. For example, using the Fixed Split, the baseline achieves a 75.9% Rank-1 accuracy (51.9% mAP) on Market-1501 and 63.6% Rank-1 (42.7% mAP) on DukeMTMC-reID. Using the Leave-One-Out split pushes performance even higher, hitting 80.5% Rank-1 on Market-1501 and 67.4% Rank-1 on DukeMTMC-reID. This simple, well-tuned baseline outperforms several recent, highly complex Unsupervised Domain Adaptation (UDA) and cross-domain methods.

Conclusion

The authors establish a strong foundation for cross-domain ReID by providing standardized multi-source splits and a powerful baseline. While fully supervised approaches still serve as an upper bound, this work facilitates fair benchmarking, which is essential for driving algorithmic improvements that can eventually close the performance gap.

Summary updated

Abstract

Person re-identification (ReID) remains a very difficult challenge in computer vision, and critical for large-scale video surveillance scenarios where an individual could appear in different camera views at different times. There has been recent interest in tackling this challenge using cross-domain approaches, which leverages data from source domains that are different than the target domain. Such approaches are more practical for real-world widespread deployment given that they don't require on-site training (as with unsupervised or domain transfer approaches) or on-site manual annotation and training (as with supervised approaches). In this study, we take a systematic approach to establishing a large baseline source domain and target domain for cross-domain person ReID. We accomplish this by conducting a comprehensive analysis to study the similarities between source domains proposed in literature, and studying the effects of incrementally increasing the size of the source domain. This allows us to establish a balanced source domain and target domain split that promotes variety in both source and target domains. Furthermore, using lessons learned from the state-of-the-art supervised person re-identification methods, we establish a strong baseline method for cross-domain person ReID. Experiments show that a source domain composed of two of the largest person ReID domains (SYSU and MSMT) performs well across six commonly-used target domains. Furthermore, we show that, surprisingly, two of the recent commonly-used domains (PRID and GRID) have too few query images to provide meaningful insights. As such, based on our findings, we propose the following balanced baseline for cross-domain person ReID consisting of: i) a fixed multi-source domain consisting of SYSU, MSMT, Airport and 3DPeS, and ii) a multi-target domain consisting of Market-1501, DukeMTMC-reID, CUHK03, PRID, GRID and VIPeR.
BibTeX
@misc{kumar2019fairest,
  author = {Devinder Kumar and Parthipan Siva and Paul Marchwica and Alexander Wong},
  title = {{Fairest of Them All: Establishing a Strong Baseline for Cross-Domain Person ReID}},
  year = {2019},
  eprint = {1907.12016},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.12016},
}
Figure