Abstract
Face clustering in large-scale datasets faces significant challenges due to "noise edges"—irrelevant connections between different identities caused by similar environmental factors like lighting or pose. This paper introduces a novel framework that enhances noise resilience through a two-stage process: prediction-driven Top-K Jaccard similarity and a Sparse Differential Transformer (SDT). By dynamically predicting the optimal number of neighbors and employing a differential attention mechanism with sparsity masking, the proposed method effectively filters out irrelevant facial relationships. Experimental results on benchmarks such as MS-Celeb-1M demonstrate state-of-the-art performance, achieving a Pairwise F-score of 95.46 on 584K images and maintaining high accuracy at scales up to 5.21M images.
Introduction
Face clustering is a fundamental task for organizing large image collections, such as "People" albums in photo management software. The process typically involves extracting facial embeddings and grouping them based on mathematical proximity. However, two primary hurdles impede performance in massive datasets. First, the "Noise Edge" problem occurs when faces of different individuals appear as nearest neighbors due to shared lighting, poses, or low image quality. Second, traditional Jaccard similarity measures often include excessive neighbors, leading to "noisy" scores that conflate distinct identities.
Methodology
The proposed framework, Diff-Cluster, addresses these issues through a two-stage pipeline designed to clean up facial relationships before clustering.
1. Prediction-driven Top-K Jaccard Similarity
Rather than utilizing a fixed number of neighbors (kNN) for every face, the model first predicts the optimal number of neighbors (Top-K) for each specific face. This allows the Jaccard similarity calculation to focus exclusively on high-purity neighbors. Additionally, the authors introduced a sigmoid-like distance transform function to amplify subtle differences between highly similar and moderately similar faces, further refining the clustering boundaries.
2. Sparse Differential Transformer (SDT)
The core of the refinement stage is the Sparse Differential Transformer. Standard Transformers often over-attend to irrelevant features in noisy graphs. To counteract this, the SDT incorporates:
- Differential Attention: Inspired by recent advancements in Large Language Models, this mechanism subtracts two separate attention maps to cancel out common noise.
- Sparsity Masking: A "Top-K mask" restricts the Transformer's attention to nodes within the predicted neighbor range, forcing the model to ignore distant, likely irrelevant connections.
- MoE-SDT: A Mixture of Experts implementation handles potential inaccuracies in Top-K prediction by considering multiple masks around the predicted K value, enhancing robustness.
Experimental Evaluation
The model was rigorously tested on several large-scale benchmarks to evaluate accuracy and scalability:
- MS-Celeb-1M: The primary face recognition benchmark. Diff-Cluster achieved a Pairwise F-score ($F_P$) of 95.46 on the 584K subset, outperforming the previous state-of-the-art (95.28). On the full 5.21M image set, it maintained a leading $F_P$ of 90.08, while traditional methods like K-Means or DBSCAN performed significantly worse.
- DeepFashion & MSMT17: To test generalization, the model was applied to clothing clustering and person re-identification. On DeepFashion, the SDT improved the $F_P$ from 38.43 to 41.76 compared to a vanilla Transformer, proving the architecture's utility beyond facial data.
Discussion and Key Insights
The success of Diff-Cluster highlights that neighbor quality is more critical than quantity. Restricting calculations to a predicted "Top-K" set significantly reduces the likelihood of merging disparate clusters. Furthermore, the Sparse Differential Transformer acts as a sophisticated denoising filter, effectively "canceling out" noise through differential subtraction.
Limitations
Despite its strengths, the framework relies on the stability of the initial Top-K prediction. While the MoE-SDT approach provides a buffer, significant errors in Stage 1 can still impact final results. Additionally, the Transformer-based architecture introduces higher computational complexity compared to simpler graph convolutional networks.
Conclusion
By integrating dynamic neighbor prediction with a noise-canceling Transformer architecture, this work sets a new standard for large-scale face clustering. The ability to maintain high purity in the face of significant environmental noise makes it a robust solution for real-world photo organization and identity management systems.
Summary updated