Unsupervised Video Person Re-Identification via Noise and Hard Frame Aware Clustering

Pengyu Xie, Xin Xu, Zheng Wang, Toshihiko Yamasaki

TL;DR

The researchers propose NHAC (Noise and Hard frame Aware Clustering), a novel unsupervised method for video-based person re-identification. By automatically trimming "noise" frames (errors/occlusions) and oversampling "hard" frames (complex poses), NHAC significantly improves clustering accuracy and feature learning, setting new state-of-the-art benchmarks on major datasets without requiring any manual labels.

What Problem Does This Paper Address?

Person Re-identification (re-ID) is the task of matching the same individual across different non-overlapping camera views. While video-based re-ID offers more data than image-based methods, most high-performing systems rely on massive amounts of manually labeled data, which is expensive and unscalable.

Unsupervised methods typically use clustering to generate "pseudo-labels" for training. However, the authors identify two major flaws in how current methods handle video tracklets:

  1. Noise Frames: Detection errors or heavy occlusions create "garbage" frames. If these are averaged into a tracklet's feature representation, they bias the data, causing the system to wrongly merge different people into the same cluster.
  2. Hard Frames: Frames with complex poses or partial occlusions are difficult to recognize but contain the most "information." Current models often fail to learn effectively from these scattered, difficult samples.

Methods

The authors introduce the NHAC framework, which builds upon Bottom-Up Clustering (BUC) but adds two critical modules:

  • Graph Trimming Module (GTM): Instead of treating a tracklet as a simple average of frames, GTM treats it as a graph. It calculates the cosine similarity between each frame and the tracklet's center. It then uses a dynamic threshold (based on the variance of the graph) to identify and prune noise frames. This ensures that the resulting tracklet feature is "clean" before clustering begins.
  • Node Re-sampling Module (NRM): To make the model more robust, the authors identify "hard" frames—those that are mathematically distant from the tracklet's average. They then oversample these hard frames during the training phase. This forces the Convolutional Neural Network (CNN) to work harder to represent these complex variations, leading to a more discriminative feature space.
  • Tracklet Triplet Loss: They implement a specific loss function that ensures frames within the same tracklet are pulled together in the feature space while being pushed away from frames in different (pseudo-labeled) clusters.

Datasets

The study evaluates NHAC on two gold-standard large-scale benchmarks:

  • DukeMTMC-VideoReID: A large-scale dataset derived from campus surveillance.
  • MARS (Motion Attribute & Re-identification Dataset): One of the largest video re-ID datasets, known for having higher quality tracklets but more "hard" frames due to pose variations.

Key Insights and Ideas

  • The Variance Matters: A fixed threshold for "noise" doesn't work because tracklets vary in length and quality across different cameras and datasets. Using a dynamic threshold based on the tracklet's own internal variance allows the system to be "noise-aware" in a flexible way.
  • Harder is Better: In deep learning, "easy" samples (clear, front-facing shots) eventually stop providing new information to the gradient. By focusing on "hard" samples (side views, bent poses), the model continues to refine its understanding of person identity.
  • Clustering Stability: The authors found that by "cleaning" the tracklets via GTM, they could prevent the "cascading error" effect where one bad noise frame causes two distinct clusters to merge prematurely.

Key Results

NHAC outperformed existing unsupervised state-of-the-art (SOTA) methods by significant margins:

  • DukeMTMC-VideoReID: Achieved 82.8% Rank-1 accuracy and 76.0% mAP (mean Average Precision). This is an 8.6-point improvement in mAP over previous SOTA methods.
  • MARS: Achieved 61.8% Rank-1 and 40.1% mAP. This represents a 10.7-point jump in mAP compared to the baseline BUC method.
  • Ablation Success: The researchers proved that GTM and NRM are complementary. On DukeMTMC, GTM alone improved Rank-1 by 2%, but adding NRM pushed that improvement to over 8%.

Limitations and Future Work

  • Iteration Peak: The authors noted that after about 14 iterations of clustering, performance begins to plateau or slightly drop. This suggests the model eventually reaches a point of "cluster collapse" or overfitting to pseudo-labels.
  • Hyperparameter Sensitivity: The performance is sensitive to the parameter $\delta$ (which controls noise relaxation). Finding the "sweet spot" for $\delta$ currently requires manual tuning for different datasets.
  • Camera Metadata: The current method does not explicitly use camera ID information. The authors suggest that incorporating camera-aware constraints could further boost performance in multi-camera environments.

Summary updated

Abstract

Unsupervised video-based person re-identification (re-ID) methods extract richer features from video tracklets than image-based ones. The state-of-the-art methods utilize clustering to obtain pseudo-labels and train the models iteratively. However, they underestimate the influence of two kinds of frames in the tracklet: 1) noise frames caused by detection errors or heavy occlusions exist in the tracklet, which may be allocated with unreliable labels during clustering; 2) the tracklet also contains hard frames caused by pose changes or partial occlusions, which are difficult to distinguish but informative. This paper proposes a Noise and Hard frame Aware Clustering (NHAC) method. NHAC consists of a graph trimming module and a node re-sampling module. The graph trimming module obtains stable graphs by removing noise frame nodes to improve the clustering accuracy. The node re-sampling module enhances the training of hard frame nodes to learn rich tracklet information. Experiments conducted on two video-based datasets demonstrate the effectiveness of the proposed NHAC under the unsupervised re-ID setting.
BibTeX
@inproceedings{xie2021unsupervised,
  author = {Pengyu Xie and Xin Xu and Zheng Wang and Toshihiko Yamasaki},
  title = {{Unsupervised Video Person Re-Identification via Noise and Hard Frame Aware Clustering}},
  booktitle = {ICME},
  year = {2021},
  doi = {10.1109/icme51207.2021.9428200},
}

Loading related content...

Figure