Enhancing Noise Resilience in Face Clustering via Sparse Differential Transformer

Dafeng Zhang, Yongqi Song, Shizhuo LiuarXiv 2025 · DOI · arXiv

Abstract

Face clustering in large-scale datasets faces significant challenges due to "noise edges"—irrelevant connections between different identities caused by similar environmental factors like lighting or pose. This paper introduces a novel framework that enhances noise resilience through a two-stage process: prediction-driven Top-K Jaccard similarity and a Sparse Differential Transformer (SDT). By dynamically predicting the optimal number of neighbors and employing a differential attention mechanism with sparsity masking, the proposed method effectively filters out irrelevant facial relationships. Experimental results on benchmarks such as MS-Celeb-1M demonstrate state-of-the-art performance, achieving a Pairwise F-score of 95.46 on 584K images and maintaining high accuracy at scales up to 5.21M images.

Introduction

Face clustering is a fundamental task for organizing large image collections, such as "People" albums in photo management software. The process typically involves extracting facial embeddings and grouping them based on mathematical proximity. However, two primary hurdles impede performance in massive datasets. First, the "Noise Edge" problem occurs when faces of different individuals appear as nearest neighbors due to shared lighting, poses, or low image quality. Second, traditional Jaccard similarity measures often include excessive neighbors, leading to "noisy" scores that conflate distinct identities.

Figure 1: Visual comparison of neighbor purity. The proposed method (SDT) effectively distinguishes between positive neighbors (blue dots) and negative noise (red triangles), whereas previous methods like FC-ESER assign high similarity to irrelevant faces.

Methodology

The proposed framework, Diff-Cluster, addresses these issues through a two-stage pipeline designed to clean up facial relationships before clustering.

Figure 3: The two-stage framework for noise-resilient face clustering. Stage 1 predicts the optimal number of neighbors (Top-K), and Stage 2 refines similarity measurements using the Sparse Differential Transformer (SDT).

1. Prediction-driven Top-K Jaccard Similarity

Rather than utilizing a fixed number of neighbors (kNN) for every face, the model first predicts the optimal number of neighbors (Top-K) for each specific face. This allows the Jaccard similarity calculation to focus exclusively on high-purity neighbors. Additionally, the authors introduced a sigmoid-like distance transform function to amplify subtle differences between highly similar and moderately similar faces, further refining the clustering boundaries.

2. Sparse Differential Transformer (SDT)

The core of the refinement stage is the Sparse Differential Transformer. Standard Transformers often over-attend to irrelevant features in noisy graphs. To counteract this, the SDT incorporates:

  • Differential Attention: Inspired by recent advancements in Large Language Models, this mechanism subtracts two separate attention maps to cancel out common noise.
  • Sparsity Masking: A "Top-K mask" restricts the Transformer's attention to nodes within the predicted neighbor range, forcing the model to ignore distant, likely irrelevant connections.
  • MoE-SDT: A Mixture of Experts implementation handles potential inaccuracies in Top-K prediction by considering multiple masks around the predicted K value, enhancing robustness.

Experimental Evaluation

The model was rigorously tested on several large-scale benchmarks to evaluate accuracy and scalability:

  • MS-Celeb-1M: The primary face recognition benchmark. Diff-Cluster achieved a Pairwise F-score ($F_P$) of 95.46 on the 584K subset, outperforming the previous state-of-the-art (95.28). On the full 5.21M image set, it maintained a leading $F_P$ of 90.08, while traditional methods like K-Means or DBSCAN performed significantly worse.
  • DeepFashion & MSMT17: To test generalization, the model was applied to clothing clustering and person re-identification. On DeepFashion, the SDT improved the $F_P$ from 38.43 to 41.76 compared to a vanilla Transformer, proving the architecture's utility beyond facial data.

Discussion and Key Insights

The success of Diff-Cluster highlights that neighbor quality is more critical than quantity. Restricting calculations to a predicted "Top-K" set significantly reduces the likelihood of merging disparate clusters. Furthermore, the Sparse Differential Transformer acts as a sophisticated denoising filter, effectively "canceling out" noise through differential subtraction.

Limitations

Despite its strengths, the framework relies on the stability of the initial Top-K prediction. While the MoE-SDT approach provides a buffer, significant errors in Stage 1 can still impact final results. Additionally, the Transformer-based architecture introduces higher computational complexity compared to simpler graph convolutional networks.

Conclusion

By integrating dynamic neighbor prediction with a noise-canceling Transformer architecture, this work sets a new standard for large-scale face clustering. The ability to maintain high purity in the face of significant environmental noise makes it a robust solution for real-world photo organization and identity management systems.

Summary updated

Abstract

The method used to measure relationships between face embeddings plays a crucial role in determining the performance of face clustering. Existing methods employ the Jaccard similarity coefficient instead of the cosine distance to enhance the measurement accuracy. However, these methods introduce too many irrelevant nodes, producing Jaccard coefficients with limited discriminative power and adversely affecting clustering performance. To address this issue, we propose a prediction-driven Top-K Jaccard similarity coefficient that enhances the purity of neighboring nodes, thereby improving the reliability of similarity measurements. Nevertheless, accurately predicting the optimal number of neighbors (Top-K) remains challenging, leading to suboptimal clustering results. To overcome this limitation, we develop a Transformer-based prediction model that examines the relationships between the central node and its neighboring nodes near the Top-K to further enhance the reliability of similarity estimation. However, vanilla Transformer, when applied to predict relationships between nodes, often introduces noise due to their overemphasis on irrelevant feature relationships. To address these challenges, we propose a Sparse Differential Transformer (SDT), instead of the vanilla Transformer, to eliminate noise and enhance the model's anti-noise capabilities. Extensive experiments on multiple datasets, such as MS-Celeb-1M, demonstrate that our approach achieves state-of-the-art (SOTA) performance, outperforming existing methods and providing a more robust solution for face clustering.
BibTeX
@misc{zhang2025enhancing,
  author = {Dafeng Zhang and Yongqi Song and Shizhuo Liu},
  title = {{Enhancing Noise Resilience in Face Clustering via Sparse Differential Transformer}},
  year = {2025},
  eprint = {2512.22612},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.22612},
}
Figure