Abstract
We study regularization in the context of small sample-size learning with over-parameterized neural networks. Specifically, we shift focus from architectural properties, such as norms on the network weights, to properties of the internal representations before a linear classifier. Specifically, we impose a topological constraint on samples drawn from the probability measure induced in that space. This provably leads to mass concentration effects around the representations of training instances, i.e., a property beneficial for generalization. By leveraging previous work to impose topological constraints in a neural network setting, we provide empirical evidence (across various vision benchmarks) to support our claim for better generalization.
Introduction
Deep neural networks are notoriously data-hungry. While they generalize surprisingly well even when over-parameterized, this success often falters in "small sample-size regimes"—scenarios where we have very few training examples per class. Standard regularization techniques usually focus on the network's weights (like $L_2$ weight decay) or the input data (like data augmentation). However, there is a third option: regularizing the internal representations—the features the network learns just before the final classification layer.
The authors argue that for a model to generalize well from few examples, the probability distribution of these features should be "densely packed" around the training instances. The challenge is how to mathematically define and enforce this "density" in a way that a neural network can actually learn. This paper introduces a novel regularization technique that uses persistent homology to "densify" these internal representations.
Methods: $\beta$-Connectedness and Mass Concentration
The authors turn to Algebraic Topology, specifically a tool called 0-dimensional Vietoris-Rips persistent homology, to solve this. They define a property called $\beta$-connectedness. Imagine your data points in the feature space. If you can connect any two points by hopping between neighbors with a maximum jump distance of $\beta$, the set is $\beta$-connected.
The core theoretical contribution is the "Mass Attracts Mass" Theorem. It proves that if a distribution is likely to produce $\beta$-connected samples, then the probability mass must concentrate around the training samples. This provides a formal link between a topological property (connectivity) and a statistical property (mass concentration) that favors generalization. Instead of asking "how smooth is the function?" (weight regularization), the authors ask "how is the probability mass distributed in the feature space?"
Topological Regularization via Persistent Homology
To enforce $\beta$-connectedness during training, the authors introduce a differentiable loss term that penalizes "death-times" in the persistent homology of a mini-batch. Persistent homology tracks how connected components merge as a distance threshold $r$ increases.
In simpler terms, the loss forces the network to map samples of the same class close enough to each other that they form a single "connected component" at a specific scale $\beta$. To make this work with standard training, they use a specific mini-batch construction where each batch contains several "sub-batches," each consisting of multiple samples from the same class. The topological loss is then applied to these sub-batches.
Interestingly, the authors found that their specific loss (penalizing deviations from $\beta$) worked better than a simpler version that only penalized distances larger than $\beta$. They hypothesize this prevents the representations from collapsing into a single point, preserving useful variance for the gradients during training.
Experimental Results
The authors evaluated their method on standard vision benchmarks modified to simulate small-sample conditions:
- MNIST-250: Only 250 total training instances.
- SVHN-250: Street View House Numbers with 250 instances.
- CIFAR10-500 / CIFAR10-1k: CIFAR10 with only 500 or 1,000 training instances.
The "Topological Regularization" (Top.-Reg.) consistently outperformed vanilla training and several state-of-the-art representation regularizers (like DeCov or Jacobian regularization).
| Dataset | Vanilla Error | Top.-Reg. (Ours) | Improvement |
|---|---|---|---|
| MNIST-250 | 7.1% | 5.6% | -1.5% |
| SVHN-250 | 30.1% | 22.5% | -7.6% |
| CIFAR10-500 | 39.4% | 36.5% | -2.9% |
| CIFAR10-1k | 29.5% | 28.5% | -1.0% |
The most dramatic gains were seen on SVHN-250, where the error dropped by over 7.5 percentage points. The authors also demonstrated that the mass concentration effect they predicted theoretically is clearly observable in the models trained with their regularizer.
Conclusion and Future Work
Measuring local density in high-dimensional spaces is notoriously difficult. Connectivity (topology) serves as a robust, differentiable proxy that captures the "shape" of the data distribution. The parameter $\beta$ acts as a "notion of scale" for the feature space and can be effectively tuned using standard cross-validation.
While promising, the method has limitations. The theoretical bounds are relative to the "mass" of the training samples; if the training set is unrepresentative, densifying around it won't help. Future work could explore higher-dimensional topological features (like "loops" or "voids") and further investigate the interplay between $\beta$ and the Lipschitz constant of the network.
Summary updated