A Topological Filter for Learning with Label Noise

Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris N. Metaxas, Chao Chen

Deep neural networks have a well-known Achilles' heel: they easily overfit to noisy labels. Because these models possess massive capacity, they tend to memorize corrupted labels—often stemming from cheap crowdsourcing, web scraping, or simple human error—which severely degrades their ability to generalize to unseen data.

This paper introduces TopoFilter, a novel method that identifies clean data by analyzing the topological structure (specifically, connected components) of data in the network's latent space. Unlike many existing heuristics, TopoFilter offers mathematical guarantees for both the purity and abundance of the filtered data.

The Problem: Label Noise and Skewed Representations

Most state-of-the-art methods attempt to filter out label noise by looking at the classifier's output probabilities. For example, they might assume that if the network is highly confident in a prediction, the label is probably correct. However, these heuristics lack solid theoretical backing and can fail unpredictably on unseen datasets or complex noise patterns.

The authors ask a fundamental question: instead of relying on final output probabilities, can we look at the spatial geometry of the data in the network's latent space to reliably separate clean data from noise?

Evolution of latent space representations from ideal to skewed, and finally the learned representation using TopoFilter.

In a perfectly learned latent space, clean data points of the same class form dense, connected clusters. However, when a network is trained on noisy data, its latent space becomes "skewed." Mislabeled data points act like isolated crumbs scattered around, and some noisy points might accidentally attach to the edges of the main clusters.

Methods: The TopoFilter Algorithm

To combat this, the authors propose an algorithm that jointly trains a neural network and filters out noisy data based on spatial topology. Instead of looking at the final softmax predictions, TopoFilter examines the penultimate layer's feature representations.

Illustration of the graph construction and peeling process, highlighting the theoretical lemmas that isolate clean data.

The algorithm works iteratively through the following steps:

  1. Feature Extraction: Extract latent features using the current state of the network.
  2. Graph Construction: Build a k-Nearest Neighbor (k-NN) graph for the data points of each class.
  3. Find the Core: Identify the largest connected component for each class, which theoretically represents the densest region of clean data.
  4. $\zeta$-filtering ("Peeling"): Because the latent space is skewed by noise, simply taking the largest cluster isn't enough. TopoFilter "peels" the boundary of the cluster by removing points that do not have a high enough fraction ($\zeta$) of neighbors sharing the same label. This trims the uncertain boundaries of the cluster.
  5. Retraining: The network is updated using only this purified, core subset of data.

Key Insights and Theoretical Guarantees

The most significant contribution of TopoFilter is its shift from probability to topology. Global structural information in the latent space is a much richer and more robust signal for detecting noise than local prediction confidence. Furthermore, the $\zeta$-filtering step is the secret sauce; by trimming the uncertain boundaries, the algorithm ensures the data pool remains uncontaminated even when the underlying feature representation is still imperfect.

Unlike many heuristic-based methods, TopoFilter comes with rigorous mathematical proofs. The authors prove two critical properties:

  • Purity: The selected data has a high probability of being clean.
  • Abundancy: The algorithm collects a large enough volume of clean data to train a strong, generalizable model.

Datasets and Key Results

The authors evaluated TopoFilter across several datasets, including CIFAR-10, CIFAR-100, Clothing1M (a massive, real-world dataset of 1 million web-scraped clothing images with ~61.5% inherent accuracy), and ModelNet40 for 3D point clouds.

  • State-of-the-Art Performance: TopoFilter consistently outperforms a wide range of baselines (including Co-teaching, MentorNet, PENCIL, and GCE) across different noise types and levels.
  • High Noise Tolerance: On CIFAR-10 with a brutal 80% uniform noise level, TopoFilter achieves 45.7% accuracy, significantly outperforming the standard model (42.0%) and baselines like Co-teaching (22.9%). Under a more moderate 60% uniform noise, it hits 80.5%.
  • Real-World Efficacy: On the Clothing1M dataset, TopoFilter achieves 74.10% accuracy, beating out strong, complex baselines like PENCIL (73.49%) and MLNT (73.47%).
  • Stable Training: Empirical analysis shows that the purity of the selected data pool remains remarkably high and stable throughout the training process, preventing the network from eventually memorizing the noise.

Limitations and Future Work

While the theoretical guarantees for purity and abundancy are strong, the authors acknowledge that their proofs currently apply to a single filtering step (one-shot). Formally proving the convergence of the entire iterative training process is left as future work.

Additionally, the current method only looks at basic connectivity (the largest connected component). Future work could leverage more advanced topological concepts, such as using a differentiable topological loss based on persistent homology, to capture even more complex geometric patterns in the latent space.

Summary updated

Abstract

Noisy labels can impair the performance of deep neural networks. To tackle this problem, in this paper, we propose a new method for filtering label noise. Unlike most existing methods relying on the posterior probability of a noisy classifier, we focus on the much richer spatial behavior of data in the latent representational space. By leveraging the high-order topological information of data, we are able to collect most of the clean data and train a high-quality model. Theoretically we prove that this topological approach is guaranteed to collect the clean data with high probability. Empirical results show that our method outperforms the state-of-the-arts and is robust to a broad spectrum of noise types and levels.
BibTeX
@inproceedings{wu2020topological,
  author = {Pengxiang Wu and Songzhu Zheng and Mayank Goswami and Dimitris N. Metaxas and Chao Chen},
  title = {{A Topological Filter for Learning with Label Noise}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=rmroNgOhnJ5},
}

Loading related content...

Figure