Why does Deep Learning work? - A perspective from Group Theory

Arnab Paul, Suresh VenkatasubramanianarXiv 2014 · OpenReview

TL;DR

This paper proposes a novel theoretical framework for understanding why deep learning works, particularly its success with unsupervised pre-training, by drawing parallels to group theory. It suggests that the layer-wise learning process in deep networks is akin to finding "simplest" features with large "stabilizers" in group theory, explaining the hierarchical emergence of complex representations.

What Problem Does This Paper Address?

Deep learning has achieved remarkable success, but a robust theoretical understanding of why it works remains elusive. Specifically, the paper tackles:

  • What representations do deep learning models capture?
  • How do higher-order representations emerge in deep networks?
  • Why is unsupervised pre-training (learning layer-by-layer) so effective?

The authors aim to bridge this gap by offering a new perspective rooted in group theory, connecting the algorithmic principles of deep learning to fundamental mathematical concepts.

Methods

The core of the paper's methodology lies in establishing an analogy between deep learning processes and group theory concepts, particularly the orbit-stabilizer theorem.

  • Group Theory Analogy: The paper posits that the space of transformations performed by a neural network can be approximated by "shadow groups." The key idea is that the pre-training step, where each layer learns to reconstruct its input (akin to finding a generative model), is equivalent to finding features that are "stabilized" by the network's transformations.
  • Orbit-Stabilizer Principle: In group theory, the orbit of an element is the set of all elements reachable from it by group actions, while the stabilizer is the subset of group elements that leave the element unchanged. The orbit-stabilizer theorem states that the size of the orbit multiplied by the size of the stabilizer equals the size of the group. The paper suggests that deep learning networks, through pre-training, implicitly search for features with large stabilizers, which, by the theorem, correspond to small orbits. Small orbits are interpreted as "simpler" features.
  • Shadow Groups: Since neural networks are not strictly groups, the authors introduce "shadow groups" as approximations that capture the essential group-like behavior of network transformations. This allows them to apply group theory principles to analyze neural networks.
  • Layer-wise Analysis: The paper extends this orbit-stabilizer perspective to multiple layers, arguing that each layer learns features with high symmetry (large stabilizers) relative to its input space. As layers deepen, these simple features combine to form increasingly complex representations.

Datasets

Not discussed in the paper. The paper focuses on theoretical framework development and uses illustrative examples rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • Deep Learning as Stabilizer Search: The paper's central insight is that unsupervised pre-training in deep learning can be viewed as a search for features that are maximally stabilized by the network's transformations. This explains why simpler features are learned first.
  • Simplicity via Stabilizers: The "simplest" features, intuitively, are those that are least affected by transformations. In group theory terms, this translates to features with large stabilizers.
  • Hierarchical Representation Emergence: The orbit-stabilizer principle, applied layer by layer, explains how complex, higher-order representations emerge. Each layer learns simple features (large stabilizers) from its input, and these combine in subsequent layers to form more intricate structures.
  • Shadow Groups as Approximations: The introduction of "shadow groups" provides a mathematical bridge, allowing the application of group theory to the non-linear, complex transformations of neural networks.
  • Connection to Generative Models: The paper links the pre-training objective (learning a generative model) directly to the group theory concept of stabilization, providing a theoretical justification for this successful deep learning paradigm.

Key Results

The paper presents theoretical results and arguments rather than quantitative performance metrics.

  • Theorem 3.1: Establishes that a neural network's action on an input figure can be consistently extended to an automorphism of the intrinsic space.
  • Theorem 4.1: Shows that neural network actions can be approximated by elements of the homeomorphism group of the intrinsic space.
  • Theorem 4.2: Demonstrates that these homeomorphisms can be locally approximated by elements of the general linear group GL(S).
  • Theorem 4.3: Establishes a bottom-up correspondence, showing that group symmetries over the intrinsic space have counterparts in neural network mappings.
  • Illustrative Examples: The paper uses examples like edges, circles, and ellipses under the action of GL2(R) to demonstrate how features with higher symmetry (like edges) are more likely to be stabilized and thus learned first.

Limitations and Future Work

  • Approximation, Not Exactness: The paper acknowledges that neural networks are not true groups, and the "shadow group" concept is an approximation. The theory applies to these approximations.
  • Supervised Learning Gap: The orbit-stabilizer principle is difficult to extend to supervised learning scenarios where input reconstruction (self-map) is not the primary objective.
  • Computational Complexity: While the theory provides a framework, performing precise calculations (e.g., defining automorphism spaces, computing stabilizer volumes) for complex moduli spaces is noted as potentially very difficult.
  • Future Work: The authors suggest exploring how this framework can be extended to supervised learning, investigating the connection between symmetry-stabilization and regularization, and further analyzing the emergent properties of higher-order representations. They also note that phenomena like adversarial examples might be explainable within this framework.

Summary updated

Abstract

Why does Deep Learning work? What representations does it capture? How do higher-order representations emerge? We study these questions from the perspective of group theory, thereby opening a new approach towards a theory of Deep learning. One factor behind the recent resurgence of the subject is a key algorithmic step called pre-training: first search for a good generative model for the input samples, and repeat the process one layer at a time. We show deeper implications of this simple principle, by establishing a connection with the interplay of orbits and stabilizers of group actions. Although the neural networks themselves may not form groups, we show the existence of {\em shadow} groups whose elements serve as close approximations. Over the shadow groups, the pre-training step, originally introduced as a mechanism to better initialize a network, becomes equivalent to a search for features with minimal orbits. Intuitively, these features are in a way the {\em simplest}. Which explains why a deep learning network learns simple features first. Next, we show how the same principle, when repeated in the deeper layers, can capture higher order representations, and why representation complexity increases as the layers get deeper.
BibTeX
@misc{paul2014why,
  author = {Arnab Paul and Suresh Venkatasubramanian},
  title = {{Why does Deep Learning work? - A perspective from Group Theory}},
  year = {2014},
  eprint = {1412.6621},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1412.6621},
}
Figure