Machine Learning

Paul Breiding, Kathlén Kohn, Bernd SturmfelsOberwolfach 2024 · DOI · OpenAlex

Abstract

One of the principal goals of machine learning is to learn in an automated way functions that represent the relationship between data points. This paper provides a rigorous geometric foundation for machine learning by treating neural networks as parametrizations of "neuromanifolds"—algebraic varieties of functions. It demonstrates how the tools of metric algebraic geometry can explain training dynamics, identify "bad" local minima, and help "learn" the underlying algebraic structures of raw data.

Introduction

While machine learning (ML) has seen massive empirical success, a deep theoretical understanding of why certain architectures work or why optimization converges to "nice" minima remains elusive. This paper addresses the gap between ML practice and mathematical theory by framing neural networks as geometric objects. It specifically tackles three areas: the expressivity of network architectures, the properties of optimization landscapes, and the "Variety Hypothesis," which posits that data points are samples from unknown algebraic varieties.

The Geometry of Neuromanifolds

The core methodology involves defining the set of all functions representable by a fixed network architecture as a "neuromanifold." For linear networks, this manifold is shown to be a determinantal variety—a set of matrices with restricted rank.

A fully connected linear neural network architecture, which can be viewed as a parametrization of a neuromanifold.

By treating architectures as algebraic varieties, we can use polynomial mapping to study complex models. For instance, linear convolutional neural networks (CNNs) are shown to be equivalent to polynomial multiplication, allowing researchers to study CNN architectures as varieties of factorizable polynomials.

Optimization Landscapes and Training Dynamics

A common myth in machine learning is that linear networks have no local minima. Geometric analysis clarifies that while global minima are often unique, "bad" (non-global) local minima are intimately tied to the singular locus of the neuromanifold. Theorem 10.2 establishes that for linear fully connected networks, the existence of non-global local minima depends entirely on the behavior of the loss function on the "smooth locus" of the neuromanifold.

A sparse network architecture representing a simpler sub-manifold within the larger neuromanifold, often associated with the singular locus.

Furthermore, the research identifies algebraic invariants of gradient flow. Proposition 10.3 shows that during training, certain matrix relationships, such as the "balancedness" of layers, remain constant. This provides a conserved quantity for the dynamical system of training, preventing numerical instability where one layer's weights explode while another's vanish.

The Variety Hypothesis and Learning from Data

The authors argue that ML is essentially trying to find a low-dimensional "variety" that explains high-dimensional data. They propose that the Euclidean Distance (ED) degree—a measure of how hard it is to find the closest point on a variety—is a crucial metric for ML performance. To learn these varieties from raw data, the paper outlines an algorithm that constructs matrices of monomials and uses Singular Value Decomposition (SVD) to find the "approximate vanishing ideal," effectively discovering the laws governing a dataset.

Conclusion and Future Work

While these geometric results provide a strong foundation for linear networks, the transition to nonlinear activations like ReLU makes the geometry much more challenging, often requiring tools from tropical geometry. Future research will focus on how the Euclidean Distance problem can be integrated into standard ML training loops and how ML can help predict heuristics for symbolic math, such as choosing monomial orderings for Gröbner bases.

Summary updated

Abstract

Abstract One of the principal goals of machine learning is to learn in an automated way functions that represent the relationship between data points.
BibTeX
@inproceedings{breiding2024machine,
  author = {Paul Breiding and Kathlén Kohn and Bernd Sturmfels},
  title = {{Machine Learning}},
  booktitle = {Oberwolfach},
  year = {2024},
  doi = {10.1007/978-3-031-51462-3\_10},
}
Figure