The Geometry of the Set of Equivalent Linear Neural Networks

Jonathan Richard Shewchuk, Sagnik BhattacharyaarXiv 2024 · DOI · arXiv

Abstract

In deep learning, the mapping from network weights to the functions they represent is many-to-one. This paper investigates the "fiber"—the set of all weight configurations yielding the same mathematical function—within linear neural networks (LNNs). By partitioning this space into smooth manifolds called "strata" based on matrix ranks, the authors provide a complete geometric and topological map of the weight space. This theoretical framework reveals how information flows through a network and offers insights into the optimization challenges posed by the geometry of weight space.

Introduction: The Complexity of the Fiber

A central focus in deep learning is the function a network learns. However, for any given function, there exist infinitely many weight configurations that produce it. This set of equivalent weights, known as a fiber, is not a simple smooth surface but an "algebraic variety" characterized by branches, sharp intersections, and varying dimensions.

Visualizing the complex algebraic variety of equivalent weight configurations.

This geometric complexity has practical implications: during training, gradient descent may encounter "spurious critical points" where progress stalls not due to functional errors, but because the specific weight configuration is mathematically "clogged." This paper aims to map this weight space rigorously for LNNs, which serve as a vital theoretical laboratory for understanding deep learning dynamics.

Methodology: Dissecting the Weight Space

The researchers utilize tools from algebraic geometry and linear algebra to analyze the fiber through several key techniques:

  • Rank Stratification: The fiber is partitioned into "strata"—smooth manifolds where the ranks of all internal matrix products remain constant.
  • Basis Flow Diagrams: The authors introduce a "Fundamental Theorem of Linear Neural Networks," decomposing layers into subspaces to track information flow and annihilation.
  • Mathematical Moves: They define "one-matrix" and "two-matrix" moves—transformations that allow navigation along the fiber without changing the output function.
  • Stratum DAGs: An algorithm was developed to generate Directed Acyclic Graphs (DAGs) illustrating the hierarchical connections between different manifolds.
Basis flow diagrams and geometric representations of rank-stratified subspaces.

Key Insights

The study yields several fundamental insights into the nature of neural network weight spaces:

  1. Topology is Rank-Driven: The "shape" of a set of equivalent weights is determined entirely by the ranks of its internal layers. Networks with identical "rank lists" possess essentially identical weight spaces under linear transformation.
  2. The Frontier Condition: Strata are hierarchically nested; if one stratum touches the boundary of another, it is entirely contained within that boundary, ensuring a highly structured geometry.
  3. Information Bottlenecks as Geometry: "Wasted" dimensions in hidden layers are identified as specific subspaces that can be "activated" by moving to a higher-dimensional stratum, providing a geometric interpretation of capacity.
  4. Navigating Training Plateaus: The defined "moves" offer a potential framework for moving a network away from training plateaus by navigating the fiber without altering the learned function.

Formal Results

The authors established several rigorous proofs and tools:

  • Dimension Formula: A closed-form formula ($D_{stratum}$) to calculate the exact dimensionality of any stratum based on its rank list.
  • Manifold Proof: Confirmation that every stratum in the rank stratification is a $C^\infty$ smooth manifold.
  • Stratum DAG Algorithm: A concrete method to enumerate all possible strata and their adjacencies for any linear architecture.
  • Geometric Equivalence: Demonstration that any two points on the same stratum are related by a bijective linear transformation.

Limitations and Future Directions

While providing a robust map for linear networks, the primary limitation is the exclusion of non-linear activation functions (e.g., ReLU), which would significantly increase the complexity of the algebraic varieties. Future work may extend these geometric principles to non-linear models and leverage the identified "moves" to develop optimization algorithms that actively avoid unfavorable geometric regions.

Summary updated

Abstract

We characterize the geometry and topology of the set of all weight vectors for which a linear neural network computes the same linear transformation $W$. This set of weight vectors is called the fiber of $W$ (under the matrix multiplication map), and it is embedded in the Euclidean weight space of all possible weight vectors. The fiber is an algebraic variety that is not necessarily a manifold. We describe a natural way to stratify the fiber--that is, to partition the algebraic variety into a finite set of manifolds of varying dimensions called strata. We call this set of strata the rank stratification. We derive the dimensions of these strata and the relationships by which they adjoin each other. Although the strata are disjoint, their closures are not. Our strata satisfy the frontier condition: if a stratum intersects the closure of another stratum, then the former stratum is a subset of the closure of the latter stratum. Each stratum is a manifold of class $C^\infty$ embedded in weight space, so it has a well-defined tangent space and normal space at every point (weight vector). We show how to determine the subspaces tangent to and normal to a specified stratum at a specified point on the stratum, and we construct elegant bases for those subspaces. To help achieve these goals, we first derive what we call a Fundamental Theorem of Linear Neural Networks, analogous to what Strang calls the Fundamental Theorem of Linear Algebra. We show how to decompose each layer of a linear neural network into a set of subspaces that show how information flows through the neural network. Each stratum of the fiber represents a different pattern by which information flows (or fails to flow) through the neural network. The topology of a stratum depends solely on this decomposition. So does its geometry, up to a linear transformation in weight space.
BibTeX
@misc{shewchuk2024geometry,
  author = {Jonathan Richard Shewchuk and Sagnik Bhattacharya},
  title = {{The Geometry of the Set of Equivalent Linear Neural Networks}},
  year = {2024},
  eprint = {2404.14855},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.14855},
}
Figure