Abstract: Deep neural networks have reshaped modern machine learning by learning powerful latent representations that often align with the manifold hypothesis: high-dimensional data lie on lower-dimensional manifolds. In this paper, we establish a connection between manifold learning and computational algebra by demonstrating how vanishing ideals can characterize the latent manifolds of deep networks. To that end, we propose a new neural architecture that (i) truncates a pretrained network at an intermediate layer, (ii) approximates each class manifold via polynomial generators of the vanishing ideal, and (iii) transforms the resulting latent space into linearly separable features through a single polynomial layer. The resulting models have significantly fewer layers than their pretrained baselines, while maintaining comparable accuracy, achieving higher throughput, and utilizing fewer parameters. Furthermore, drawing on spectral complexity analysis, we derive sharper theoretical guarantees for generalization, showing that our approach can in principle offer tighter bounds than standard deep networks. Numerical experiments confirm the effectiveness and efficiency of the proposed approach.
1. Introduction
Deep Neural Networks (DNNs) are traditionally viewed as engines that learn to disentangle complex data manifolds through successive non-linear transformations. While effective, this "unfolding" process typically requires deep architectures with millions of parameters, leading to high computational costs and limited theoretical transparency. This paper, presented at ICML 2025, introduces Vanishing Ideal Networks (VI-Nets), a framework that leverages computational algebra to characterize these latent manifolds directly. By replacing the deep "tail" of a network with algebraically-derived polynomial features, VI-Nets offer a path toward more efficient and interpretable architectures.
2. The Manifold Hypothesis and Vanishing Ideals
The manifold hypothesis posits that high-dimensional data, such as natural images, reside on much lower-dimensional surfaces (manifolds) embedded within the high-dimensional space. In a classification context, a DNN's goal is to transform these manifolds until they are linearly separable.
VI-Nets approach this problem by treating class manifolds as algebraic varieties—sets defined by the common zeros of a collection of polynomials. The set of all polynomials that evaluate to zero on a given set of points is known as the vanishing ideal. By identifying the generators of this ideal, we can algebraically define the "surface" each class lives on in the latent space. If we can capture this structure at an intermediate stage, the remaining deep layers of a standard network may become redundant.
3. The VI-Net Architecture
The construction of a VI-Net follows a three-step pipeline:
- Truncation: A pretrained network (such as a ResNet) is cut off at an intermediate layer. This layer provides a latent representation where the manifolds are partially unfolded but not yet linearly separable.
- Algebraic Characterization: Latent features are extracted at the truncation point. For each class, algorithms such as Oracle-based Approximate Vanishing Ideal (OAVI) or the Approximate Buchberger-Möller (ABM) algorithm are used to find a set of polynomials that are approximately zero for that class's data points but non-zero for others.
- Polynomial Layer: The discarded layers of the original network are replaced with a single layer that evaluates these polynomials. This transforms the latent space into a new feature space where classes are linearly separable, allowing for classification with a simple linear head.
Scaling to Modern Datasets
To make this algebraic approach viable for large-scale deep learning, the authors introduced several optimizations:
- Dimensionality Reduction: PCA is applied to latent features to prevent the number of polynomial terms from exploding.
- Stochastic Algorithms: New stochastic variants of vanishing ideal algorithms allow the model to process data subsets efficiently, saving memory.
- Discriminative Pruning: Polynomials are ranked by their ability to vanish on a target class while remaining large on others, ensuring only the most useful features are retained.
4. Theoretical Foundations
Beyond empirical efficiency, VI-Nets offer improved theoretical properties. Using the spectral complexity framework, the authors prove that because their polynomial layers are sparse and highly structured, VI-Nets can theoretically offer tighter bounds on generalization error than standard deep networks. This suggests that the algebraic structure acts as a powerful regularizer, capturing the essential geometry of the data without the noise often found in overparameterized deep layers.
5. Empirical Results
The approach was evaluated using standard image classification benchmarks, including CIFAR-10 (using ResNet-18) and CIFAR-100 (using ResNet-34).
- Efficiency and Accuracy: On CIFAR-100, a "VI-Net-Small" configuration—which removes approximately 50% of ResNet-34's layers—achieved 74.03% accuracy, nearly matching the full ResNet-34 baseline of 74.75%.
- Parameter Savings: VI-Nets significantly reduced parameter counts. For example, the VI-Net-Tiny version for ResNet-34 reduced the total parameters from ~20M to ~3.5M.
- Throughput: By reducing depth, VI-Nets achieved higher throughput, processing images 1.5x to 2x faster than the original baseline models in certain configurations.
6. Conclusion and Future Work
VI-Nets demonstrate that bridging deep learning with computational algebra can lead to models that are faster, smaller, and more theoretically grounded. While the current method relies on pretrained networks to provide a latent space, it proves that vanishing ideal generators capture complex geometric information that linear models cannot. Future research may explore training these algebraic structures from scratch or extending the framework to Transformer-based architectures.
Summary updated