Topologically Densified Distributions

Christoph D. Hofer, Florian Graf, Marc Niethammer, Roland Kwitt

Introduction

Deep learning models are notoriously data-hungry. While over-parameterized networks generalize surprisingly well in high-data regimes, they struggle significantly when training samples are scarce. Standard solutions usually involve regularizing the network weights (like weight decay) or augmenting the input data. However, these methods don't directly address the "shape" of the information the network learns.

This paper introduces a novel regularization technique that improves neural network generalization in "small data" regimes by constraining the topology of internal representations. By forcing the representations of training samples to be "connected" at a specific scale, the authors provably show that the underlying probability distribution concentrates around these samples, leading to better performance on unseen data.

The Challenge of Fragmented Representations

For a model to generalize well, the probability mass in the internal representation space (the layer just before the final classifier) should be concentrated around the training instances. If the network's internal "map" is scattered or fragmented, it is more likely to misclassify a slightly different version of a known object. The core mathematical challenge is defining and enforcing this "clumping" or concentration of probability mass without collapsing the representations entirely.

Topological Regularization via Persistent Homology

To solve this, the authors turn to Algebraic Topology, specifically a tool called 0-dimensional persistent homology.

The Intuition: Imagine the representations of your training images as points in space. If you grow a ball of radius $r$ around every point, at what value of $r$ do all points from the same class merge into a single connected component?

0-dimensional persistent homology tracks the 'death times' as points merge into connected components with increasing radius $r$.

$\beta$-connectedness: The authors define a set of points as $\beta$-connected if they all merge together before the radius reaches a threshold $\beta$. In graph theory terms, this is equivalent to saying the longest edge in the Minimum Spanning Tree (MST) of these points is shorter than $\beta$.

A regularized representation space where training samples are $\beta$-connected, forcing the probability mass to concentrate.

The Loss Function: They introduce a topological regularization loss term that penalizes the network if the "death times" (the radii at which components merge) deviate from $\beta$: $$ \mathcal{L}(\mathbb{B}) = \sum |d - \beta| $$ This forces the network to learn a feature extractor $\phi$ that maps training samples of the same class into a "densified" or well-connected cluster in the representation space. Unlike standard weight regularization, this acts on the push-forward probability measure—the distribution of data after it has been processed by the network's layers.

Key Insights

  • Mass Attracts Mass: The core theoretical contribution is proving that if you force a finite sample to be $\beta$-connected, it provides a lower bound on the probability mass of the "$\epsilon$-extension" of those points. In plain English: forcing the training points to be close together topologically forces the entire distribution to concentrate around them.
  • Shifting Focus from Weights to Representations: Most regularization (like Dropout or $L_2$) focuses on the capacity of the network. This work focuses on the utility of the representations, ensuring the network organizes its internal "thoughts" in a way that favors generalization.
  • Differentiable Topology: By leveraging recent advances in persistent homology, the authors make a seemingly discrete geometric property (connectedness) fully differentiable, allowing it to be optimized via standard Stochastic Gradient Descent (SGD).

Empirical Results

The researchers evaluated their method on three standard vision benchmarks, specifically modified for small sample-size regimes (e.g., MNIST-250, SVHN-250, and CIFAR10-500). These small subsets are much harder than the full datasets and serve as a rigorous test for regularization effectiveness.

Topological Regularization consistently outperformed vanilla models and other state-of-the-art representation regularizers (like DeCov or Jacobian regularization):

  • SVHN-250: The error rate dropped from 30.1% (Vanilla) to 22.5%.
  • CIFAR10-500: The error rate dropped from 39.4% (Vanilla) to 36.5%.
  • MNIST-250: The error rate dropped from 7.1% to 5.6%.

Importantly, the hyperparameter $\beta$ proved robust and could be effectively chosen using a very small validation set, making it practical for real-world use where data is limited.

Limitations and Future Directions

While highly effective, the approach has limitations. Computing persistent homology (or MSTs) for every mini-batch adds computational overhead compared to simple weight decay, necessitating the use of sub-batches. Furthermore, while it works well for the 128-dimensional representation space used in the study, the behavior of topological constraints in much higher-dimensional manifolds remains an open question.

Future work may explore preventing "lifetimes" from collapsing entirely (keeping them near $\beta$ rather than just below it) to preserve gradient variance, potentially offering deeper insights into the optimization process.

Summary updated

Abstract

We study regularization in the context of small sample-size learning with over-parametrized neural networks. Specifically, we shift focus from architectural properties, such as norms on the network...
BibTeX
@inproceedings{hofer2020topologically,
  author = {Christoph D. Hofer and Florian Graf and Marc Niethammer and Roland Kwitt},
  title = {{Topologically Densified Distributions}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=qOwSJTFuHcA},
}
Figure