Approximating Latent Manifolds in Neural Networks via Vanishing Ideals

Nico Pelleriti, Max Zimmer, Elias Wirth, Sebastian Pokutta

This paper presents a fascinating bridge between the worlds of deep learning and computational algebra. The authors introduce VI-Nets, a method to simplify and speed up deep neural networks by replacing their final layers with a single polynomial layer derived from the "vanishing ideals" of class-specific data manifolds.

TL;DR

VI-Nets truncate pretrained neural networks and replace their deep, parameter-heavy final layers with a single polynomial layer that captures the algebraic structure of class manifolds. This approach significantly reduces parameter counts and increases throughput while maintaining competitive accuracy and providing tighter theoretical generalization bounds.

What Problem Does This Paper Address?

Deep neural networks (NNs) are often criticized for being "black boxes" with massive parameter counts. A core theoretical justification for their success is the manifold hypothesis: the idea that high-dimensional real-world data actually lies on much lower-dimensional manifolds. While deep NNs are great at "disentangling" these manifolds to make classes linearly separable in the final layer, they require many layers and millions of parameters to do so.

The authors ask: Can we use the geometric and algebraic structure of these manifolds to simplify the network?

Specifically, they tackle the challenge of using vanishing ideals—a concept from algebraic geometry—to characterize these manifolds. A vanishing ideal is the set of all polynomials that evaluate to zero on a specific set of points. Historically, algorithms to find these polynomials haven't scaled to the high-dimensional latent spaces of modern NNs.

Methods

The authors propose a pipeline called the VI-Net Pipeline (Algorithm 1):

  1. Truncation: Take a pretrained network (like a ResNet) and cut it off at an intermediate layer.
  2. Dimensionality Reduction: Use PCA to project the high-dimensional latent features into a manageable space (e.g., 128 dimensions).
  3. Manifold Approximation: For each class, use vanishing ideal algorithms (like ABM or OAVI) to find a set of polynomial "generators." These are polynomials that are (nearly) zero for samples of that class but non-zero for others.
  4. Pruning & Sparsification: To keep the model efficient, they introduce a pruning score to select only the most discriminative polynomials and use stochastic optimization to ensure the polynomials are "coefficient-sparse" (having few non-zero terms).
  5. Polynomial Layer: The truncated network is appended with a layer that evaluates these polynomials. Because the polynomials "vanish" on their respective classes, they act as powerful nonlinear features that make the data easily separable by a simple final linear head.
Figure 1. Comparison between a baseline NN (left) and a truncated NN (right). The baseline NN output is interpreted as class proba- bilities. The truncated NN output is interpreted using the manifold hypothesis, suggesting that classes lie on distinct manifolds. We characterize these manifolds using a set of polynomials, capturing their underlying structure. The depicted manifold is a visualization of a Calabi–Yau manifold, licensed under the Creative Commons Attribution-Share Alike 2.5 Generic license (Lunch, 2007).

Datasets

The researchers evaluated their approach using standard image classification benchmarks:

  • CIFAR-10
  • CIFAR-100 Both were tested using ResNet-18 and ResNet-34 as the baseline architectures.

Key Insights and Ideas

  • Polynomial Separability: The authors demonstrate that even if features at an intermediate layer aren't linearly separable, they are often polynomially separable. By explicitly finding these polynomials, you can skip the remaining deep layers of the original network.
  • Algebraic Characterization: By treating the latent space as an algebraic variety (a set defined by polynomial roots), they provide a more structured way to capture data relationships than standard "black box" layers.
  • Generalization Guarantees: Using spectral complexity analysis, they prove that VI-Nets can offer tighter generalization bounds than standard deep networks. Essentially, because the polynomial layer is highly structured and sparse, the model is less likely to overfit.
  • Efficiency through Sparsity: The method relies on finding sparse polynomials. As shown in Figure 2, by pruning non-essential monomials, they drastically reduce the computational cost of evaluating the polynomial layer.
Illustration of parameter-efficient representation through pruning

Key Results

  • Parameter Efficiency: VI-Nets achieve massive compression. For ResNet-18 on CIFAR-10, the "Tiny" variant uses only 1.86M parameters compared to the baseline's 11.24M, while maintaining an accuracy of 88.62% (vs 92.89%).
  • Higher Throughput: Because they remove many convolutional layers, VI-Nets are faster. The VI-Net18-Tiny achieved ~100,000 images/sec, roughly 1.5x faster than the standard ResNet-18.
  • Robustness to Truncation: As shown in Figure 3, while a simple linear head fails quickly as you remove more layers, the VI-Net maintains high accuracy even after removing 50% or more of the original network's layers.
Figure 3. CIFAR-100 (ResNet-34): Test accuracy vs. percentage of removed convolutional layers. As more layers are removed, a classifier with only a linear head after truncation experiences a sharp performance drop, whereas VI-Net maintains accuracy.

Limitations and Future Work

  • Manifold Overlap: If the latent manifolds of different classes overlap significantly, polynomials may fail to accurately distinguish them.
  • Intrinsic Dimensionality: If the data manifolds have very high intrinsic dimensionality, the number of required polynomial terms might explode, negating the efficiency gains.
  • Architecture Scope: This study focused on CNNs. The authors suggest that exploring the latent spaces of Transformer models is a promising next step.
  • Scaling: While they introduced stochastic adaptations, computing vanishing ideals for extremely large-scale datasets remains a computational challenge.

Summary updated

Abstract

Deep neural networks have reshaped modern machine learning by learning powerful latent representations that often align with the manifold hypothesis: high-dimensional data lie on lower-dimensional manifolds. In this paper, we establish a connection between manifold learning and computational algebra by demonstrating how vanishing ideals can characterize the latent manifolds of deep networks. To that end, we propose a new neural architecture that (i) truncates a pretrained network at an intermediate layer, (ii) approximates each class manifold via polynomial generators of the vanishing ideal, and (iii) transforms the resulting latent space into linearly separable features through a single polynomial layer. The resulting models have significantly fewer layers than their pretrained baselines, while maintaining comparable accuracy, achieving higher throughput, and utilizing fewer parameters. Furthermore, drawing on spectral complexity analysis, we derive sharper theoretical guarantees for generalization, showing that our approach can in principle offer tighter bounds than standard deep networks. Numerical experiments confirm the effectiveness and efficiency of the proposed approach.
BibTeX
@inproceedings{pelleriti2025approximating,
  author = {Nico Pelleriti and Max Zimmer and Elias Wirth and Sebastian Pokutta},
  title = {{Approximating Latent Manifolds in Neural Networks via Vanishing Ideals}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=WYlerYGDPL},
}
Figure