Abstract
In deep learning, the mapping from network weights to the functions they represent is many-to-one. This paper investigates the "fiber"—the set of all weight configurations yielding the same mathematical function—within linear neural networks (LNNs). By partitioning this space into smooth manifolds called "strata" based on matrix ranks, the authors provide a complete geometric and topological map of the weight space. This theoretical framework reveals how information flows through a network and offers insights into the optimization challenges posed by the geometry of weight space.
Introduction: The Complexity of the Fiber
A central focus in deep learning is the function a network learns. However, for any given function, there exist infinitely many weight configurations that produce it. This set of equivalent weights, known as a fiber, is not a simple smooth surface but an "algebraic variety" characterized by branches, sharp intersections, and varying dimensions.
This geometric complexity has practical implications: during training, gradient descent may encounter "spurious critical points" where progress stalls not due to functional errors, but because the specific weight configuration is mathematically "clogged." This paper aims to map this weight space rigorously for LNNs, which serve as a vital theoretical laboratory for understanding deep learning dynamics.
Methodology: Dissecting the Weight Space
The researchers utilize tools from algebraic geometry and linear algebra to analyze the fiber through several key techniques:
- Rank Stratification: The fiber is partitioned into "strata"—smooth manifolds where the ranks of all internal matrix products remain constant.
- Basis Flow Diagrams: The authors introduce a "Fundamental Theorem of Linear Neural Networks," decomposing layers into subspaces to track information flow and annihilation.
- Mathematical Moves: They define "one-matrix" and "two-matrix" moves—transformations that allow navigation along the fiber without changing the output function.
- Stratum DAGs: An algorithm was developed to generate Directed Acyclic Graphs (DAGs) illustrating the hierarchical connections between different manifolds.
Key Insights
The study yields several fundamental insights into the nature of neural network weight spaces:
- Topology is Rank-Driven: The "shape" of a set of equivalent weights is determined entirely by the ranks of its internal layers. Networks with identical "rank lists" possess essentially identical weight spaces under linear transformation.
- The Frontier Condition: Strata are hierarchically nested; if one stratum touches the boundary of another, it is entirely contained within that boundary, ensuring a highly structured geometry.
- Information Bottlenecks as Geometry: "Wasted" dimensions in hidden layers are identified as specific subspaces that can be "activated" by moving to a higher-dimensional stratum, providing a geometric interpretation of capacity.
- Navigating Training Plateaus: The defined "moves" offer a potential framework for moving a network away from training plateaus by navigating the fiber without altering the learned function.
Formal Results
The authors established several rigorous proofs and tools:
- Dimension Formula: A closed-form formula ($D_{stratum}$) to calculate the exact dimensionality of any stratum based on its rank list.
- Manifold Proof: Confirmation that every stratum in the rank stratification is a $C^\infty$ smooth manifold.
- Stratum DAG Algorithm: A concrete method to enumerate all possible strata and their adjacencies for any linear architecture.
- Geometric Equivalence: Demonstration that any two points on the same stratum are related by a bijective linear transformation.
Limitations and Future Directions
While providing a robust map for linear networks, the primary limitation is the exclusion of non-linear activation functions (e.g., ReLU), which would significantly increase the complexity of the algebraic varieties. Future work may extend these geometric principles to non-linear models and leverage the identified "moves" to develop optimization algorithms that actively avoid unfavorable geometric regions.
Summary updated