Position: $C^*$-Algebraic Machine Learning $-$ Moving in a New Direction

Yuka Hashimoto, Masahiro Ikeda, Hachem Kadri

Modern machine learning faces two major hurdles: increasingly complex structured data (like graphs, time-series, and functional data) and the explosion of large-scale multi-model systems (like ensembles, multitask learning, and meta-learning).

Standard ML methods typically rely on real or complex numbers. While effective, this "scalar-centric" view often forces us to flatten or simplify data, losing rich structural information. Furthermore, managing thousands of separate models for different tasks is computationally and conceptually messy.

To address this, researchers propose a paradigm shift: $C^*$-algebraic Machine Learning. By replacing scalar values with elements of a $C^*$-algebra (such as matrices or functions), this framework provides a unified mathematical language capable of preserving structural information and naturally handling complex model interactions.

Overview of C*-algebraic machine learning

The $C^*$-Algebraic Framework

The paper introduces the cross-fertilization of $C^*$-algebra and machine learning through two primary lenses:

1. Kernel Methods (RKHM)

The authors generalize the well-known Reproducing Kernel Hilbert Space (RKHS) into a Reproducing Kernel Hilbert $C^*$-module (RKHM). Instead of a kernel outputting a single scalar number, it outputs an element of a $C^*$-algebra. This allows the model to output entire functions or operators, which is ideal for predicting structured outputs like spectral measures in quantum mechanics or functional responses in time-series analysis.

2. $C^*$-algebraic Neural Networks

They propose neural networks where weights and biases are elements of a $C^*$-algebra rather than simple scalars. For example, if the algebra is the space of continuous functions $C(Z)$, a single $C^*$-algebraic network can represent an infinite ensemble of standard networks indexed by a variable $z \in Z$.

Comparison of standard multiple models vs. a single C*-algebraic model

Because $C^*$-algebras encompass complex numbers, matrices, and bounded linear operators, this framework provides a single set of tools to analyze everything from simple linear regression to complex quantum states.

Key Insights and Theoretical Results

As a position paper, the focus is on establishing mathematical foundations rather than empirical evaluations on standard benchmarks. Key theoretical findings include:

  • Deceptive Linearity: One of the most surprising findings is that even if a $C^*$-algebraic neural network uses linear activation functions, the resulting model is non-linear (specifically, a polynomial) with respect to the algebra's internal structure. The paper proves that a $C^*$-algebra net of depth $L$ acts as a degree $L$ polynomial. This suggests that $C^*$-algebraic nets might achieve high expressivity with simpler architectures.
  • Bridging the Convexity Gap: Neural network optimization is notoriously non-convex. However, the authors prove that by "averaging" $C^*$-algebraic nets (integrating over the algebra), the optimization problem for learning the resulting probability measure becomes convex (provided the loss function is continuous and convex). This offers a theoretical path toward more stable and global optimization in deep learning.
  • Milder Dimensionality Dependency: By using the operator norm inherent in $C^*$-algebras rather than standard Euclidean norms, generalization error bounds can potentially be made less dependent on the raw dimension of the output data.
  • The Quantum Bridge: $C^*$-algebra was originally developed for quantum mechanics. By adopting this framework, ML researchers gain immediate access to decades of rigorous mathematical physics, potentially leading to more efficient "quantum-native" machine learning algorithms.

Limitations and Future Work

While theoretically promising, $C^*$-algebraic ML faces several practical hurdles:

  • Computational Complexity: Representing elements of infinite-dimensional algebras requires discretization (e.g., using Fourier series). If the number of points for discretization is large, the computational cost—especially for Gram matrices in kernel methods—becomes prohibitive.
  • Lack of Inverses: Unlike real numbers, elements in a $C^*$-algebra do not always have an inverse. This makes directly porting some classical algorithms (like those requiring matrix inversion) non-trivial.
  • Software Ecosystem: There is currently no standard software library (like a "$C^*$-PyTorch") for these methods. Developing such tools is a necessary step for community adoption.

Looking forward, the authors suggest exploring Cuntz algebras for variable-length data (like text) and Approximately Finite (AF) dimensional algebras for data with varying dimensions (like social network graphs).

Summary updated

Abstract

Machine learning has a long collaborative tradition with several fields of mathematics, such as statistics, probability and linear algebra. We propose a new direction for machine learning research: $C^*$-algebraic ML $-$ a cross-fertilization between $C^*$-algebra and machine learning. The mathematical concept of $C^*$-algebra is a natural generalization of the space of complex numbers. It enables us to unify existing learning strategies, and construct a new framework for more diverse and information-rich data models. We explain why and how to use $C^*$-algebras in machine learning, and provide technical considerations that go into the design of $C^*$-algebraic learning models in the contexts of kernel methods and neural networks. Furthermore, we discuss open questions and challenges in $C^*$-algebraic ML and give our thoughts for future development and applications.
BibTeX
@inproceedings{hashimoto2024position,
  author = {Yuka Hashimoto and Masahiro Ikeda and Hachem Kadri},
  title = {{Position: $C^*$-Algebraic Machine Learning $-$ Moving in a New Direction}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=w9nxTXuaCc},
}
Figure