Holomorphic feedforward networks

Michael R. Douglas

TL;DR

This paper introduces "holomorphic feedforward networks" (FFNs) that bridge machine learning and complex geometry. By framing FFNs as representations of holomorphic line bundles on complex manifolds, the authors explore their function approximation capabilities and draw parallels between machine learning's supervised learning and the problem of finding Ricci-flat Kähler metrics.

What Problem Does This Paper Address?

The paper tackles the "curse of dimensionality," a fundamental challenge in both machine learning and physics, particularly in approximating functions in high-dimensional spaces. In machine learning, this manifests as needing an exponentially large number of parameters to represent complex functions. In physics, it appears in problems like finding Ricci-flat Kähler metrics on Calabi-Yau manifolds, which are crucial in string theory. The authors aim to leverage the successes of deep learning in mitigating this curse and simplifying optimization to address these geometric problems.

Methods

The core innovation is the introduction of "holomorphic feedforward networks" (FFNs). These networks are designed to represent sections of holomorphic line bundles on complex manifolds. The authors adapt the standard FFN architecture, which typically involves linear transformations and nonlinear activation functions, by making these components correspond to geometric operations. Specifically, the linear transformations are interpreted as holomorphic maps between complex vector spaces, and the activation functions are chosen to have geometric interpretations, such as exponentiation of sections.

The paper also explores two specific types of these networks:

  • Multilayer holomorphic embeddings: These networks are used to define parameterized maps from a manifold $M$ into projective space, which in turn define parameterized Kähler metrics.
  • Bihomogeneous networks: A variation that first forms sesquilinear combinations of holomorphic sections before applying the nonlinear network construction. This approach aims to reduce the number of parameters while retaining the ability to represent complex structures.

The authors also draw formal parallels between supervised learning and the problem of finding numerical Ricci-flat Kähler metrics, suggesting that techniques from one field might inform the other.

Datasets

Not discussed in the paper. The paper focuses on theoretical frameworks and numerical methods rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • Bridging ML and Geometry: The paper establishes a formal connection between the structure of feedforward neural networks and geometric objects like holomorphic line bundles and Kähler metrics. This opens up new avenues for applying machine learning techniques to complex geometric problems.
  • Holomorphic Networks as Geometric Tools: By designing FFNs with specific geometric interpretations for their layers and activation functions, the authors create a powerful tool for parameterizing and approximating geometric objects.
  • Curse of Dimensionality Mitigation: The paper hypothesizes that, similar to deep learning, these holomorphic networks can mitigate the curse of dimensionality in geometric problems by allowing for the representation of complex functions with fewer parameters than traditional methods.
  • Formal Parallels: The authors highlight striking similarities between the optimization problems in supervised learning (minimizing loss functions) and finding Ricci-flat Kähler metrics, suggesting that insights from one domain can be transferred to the other.
  • Potential for Numerical Ricci-Flat Metrics: The work lays the groundwork for using these networks to find numerical Ricci-flat Kähler metrics, a computationally challenging problem in theoretical physics.

Key Results

The paper primarily presents a theoretical framework and introduces new classes of parameterized functions (holomorphic and bihomogeneous networks). It does not present specific quantitative results from numerical experiments in this particular document, but refers to related work ([16]) where numerical evidence for the effectiveness of these metrics in representing Ricci-flat metrics is discussed. The paper suggests that bihomogeneous networks are better behaved than holomorphic networks for small intermediate widths and can represent structures on short length scales with fewer parameters.

Limitations and Future Work

  • Nonconvex Optimization: The paper acknowledges that the loss functions associated with these networks can be nonconvex, which is a common challenge in machine learning. However, it notes that this has not been a serious problem in practice for ML applications and suggests that more geometric definitions of these networks might avoid this issue.
  • Accuracy of Approximation: A significant portion of the paper is dedicated to posing precise questions about the accuracy of approximation achieved by these networks, particularly concerning the role of network depth versus the number of parameters. The authors speculate on how the curse of dimensionality might be broken or mitigated in specific cases.
  • Generalization of Conjectures: The paper proposes strong conjectures about the error bounds for approximating general metrics and suggests that these bounds might hold even when the choice of inputs depends on the function being approximated.
  • Further Exploration of Bihomogeneous Networks: The authors mention that they have not yet tried the matrix product state construction for bihomogeneous networks, indicating a potential area for future research.
  • Geometric Definitions: The paper calls for more geometric definitions of these networks to potentially improve the optimization landscape and provide a more unambiguous definition of balanced bihomogeneous metrics.

Summary updated

Abstract

A very popular model in machine learning is the feedforward neural network (FFN). The FFN can approximate general functions and mitigate the curse of dimensionality. Here we introduce FFNs which represent sections of holomorphic line bundles on complex manifolds, and ask some questions about their approximating power. We also explain formal similarities between the standard approach to supervised learning and the problem of finding numerical Ricci flat Kähler metrics, which allow carrying some ideas between the two problems.
BibTeX
@inproceedings{douglas2022holomorphic,
  author = {Michael R. Douglas},
  title = {{Holomorphic feedforward networks}},
  booktitle = {Pure and Applied Mathematics Quarterly},
  year = {2022},
  doi = {10.4310/pamq.2022.v18.n1.a7},
}
Figure