Abstract
One of the principal goals of machine learning is to learn in an automated way functions that represent the relationship between data points. This paper provides a rigorous geometric foundation for machine learning by treating neural networks as parametrizations of "neuromanifolds"—algebraic varieties of functions. It demonstrates how the tools of metric algebraic geometry can explain training dynamics, identify "bad" local minima, and help "learn" the underlying algebraic structures of raw data.
Introduction
While machine learning (ML) has seen massive empirical success, a deep theoretical understanding of why certain architectures work or why optimization converges to "nice" minima remains elusive. This paper addresses the gap between ML practice and mathematical theory by framing neural networks as geometric objects. It specifically tackles three areas: the expressivity of network architectures, the properties of optimization landscapes, and the "Variety Hypothesis," which posits that data points are samples from unknown algebraic varieties.
The Geometry of Neuromanifolds
The core methodology involves defining the set of all functions representable by a fixed network architecture as a "neuromanifold." For linear networks, this manifold is shown to be a determinantal variety—a set of matrices with restricted rank.
By treating architectures as algebraic varieties, we can use polynomial mapping to study complex models. For instance, linear convolutional neural networks (CNNs) are shown to be equivalent to polynomial multiplication, allowing researchers to study CNN architectures as varieties of factorizable polynomials.
Optimization Landscapes and Training Dynamics
A common myth in machine learning is that linear networks have no local minima. Geometric analysis clarifies that while global minima are often unique, "bad" (non-global) local minima are intimately tied to the singular locus of the neuromanifold. Theorem 10.2 establishes that for linear fully connected networks, the existence of non-global local minima depends entirely on the behavior of the loss function on the "smooth locus" of the neuromanifold.
Furthermore, the research identifies algebraic invariants of gradient flow. Proposition 10.3 shows that during training, certain matrix relationships, such as the "balancedness" of layers, remain constant. This provides a conserved quantity for the dynamical system of training, preventing numerical instability where one layer's weights explode while another's vanish.
The Variety Hypothesis and Learning from Data
The authors argue that ML is essentially trying to find a low-dimensional "variety" that explains high-dimensional data. They propose that the Euclidean Distance (ED) degree—a measure of how hard it is to find the closest point on a variety—is a crucial metric for ML performance. To learn these varieties from raw data, the paper outlines an algorithm that constructs matrices of monomials and uses Singular Value Decomposition (SVD) to find the "approximate vanishing ideal," effectively discovering the laws governing a dataset.
Conclusion and Future Work
While these geometric results provide a strong foundation for linear networks, the transition to nonlinear activations like ReLU makes the geometry much more challenging, often requiring tools from tropical geometry. Future research will focus on how the Euclidean Distance problem can be integrated into standard ML training loops and how ML can help predict heuristics for symbolic math, such as choosing monomial orderings for Gröbner bases.
Summary updated