Mini-Workshop: Mathematics of Entropic AI in the Natural Sciences

Susanne Gerber, Illia Horenko, Rupert Klein, T. OkaneOberwolfach 2024 · DOI

Abstract

The mathematical framework of approximate entropic learning provides a robust and efficient alternative to traditional machine learning in "small data" regimes. These regimes, characterized by high-dimensional problem spaces and limited statistical samples, are prevalent across the natural sciences—including geosciences, climate research, economics, and biomedicine. This report summarizes the findings of a 2024 mini-workshop at the Mathematisches Forschungsinstitut Oberwolfach, which brought together experts to establish a common mathematical framework for entropic AI, aiming to bolster methodological development and provide rigorous theoretical foundations for these emergent techniques.

Introduction: Beyond Data-Hungry Models

While Deep Learning (DL) has achieved remarkable success in domains with massive datasets, such as computer vision and natural language processing, its application in the natural sciences is often hindered by the "small data" challenge. In fields like climate modeling or biomedicine, researchers frequently encounter underdetermined problems where the number of features far exceeds the available samples. In these contexts, traditional Deep Neural Networks (DNNs) often fail due to their lack of interpretability, high computational costs, and susceptibility to adversarial perturbations.

To address these limitations, researchers are developing "Entropic AI"—a framework that leverages information theory to create models that are not only more efficient but also physically interpretable.

The Small Data Challenge in Scientific Computing

In the natural sciences, the data-hungry nature of DL presents several critical barriers:

  1. Interpretability: DNNs often function as "black boxes," making it difficult for scientists to derive or verify underlying physical laws.
  2. Data Scarcity: Rare events, such as specific climate phenomena or unique biological mutations, do not provide the volume of data required for standard training.
  3. Computational and Environmental Costs: The carbon footprint and economic barriers of training massive models are increasingly unsustainable.
  4. Robustness: DNNs are vulnerable to "adversarial attacks," where minor input perturbations lead to significant errors, a risk that is unacceptable in critical scientific applications.

Methodological Innovations: eSPA+ and Entropy Regularization

The workshop highlighted Sparse Probabilistic Approximations (SPA) and specifically eSPA+ (entropy-optimal Scalable Probabilistic Approximation) as promising solutions. Unlike DNNs, which rely on layers of weights and non-linear activations, eSPA+ treats learning as a constrained optimization problem.

Key Technical Features:

  • Entropy Regularization: By using Shannon entropy as a penalty term, the model identifies the "broadest" probability distribution that fits the data. This prevents overfitting to noise, a common issue in small datasets.
  • Dimension Reduction: The framework incorporates feature weighting ($W$) to automatically perform feature selection during training, identifying the primary drivers of a given outcome.
  • Newton-type Solvers: The mathematical structure of eSPA+ allows for second-order optimization methods. These converge significantly faster than the Stochastic Gradient Descent (SGD) typically used in DL.
  • Physical Reservoir Computing: Beyond software, researchers explored "in-materio" computing using magnetic skyrmions—topologically stable magnetic whirls—to process information with extreme energy efficiency.

Empirical Results and Insights

The efficacy of these methods was validated across several diverse benchmarks:

Climate Modeling and ENSO Prediction

In predicting El Niño/La Niña (ENSO) events using sea surface temperature data, the eSPA algorithm outperformed traditional Long Short-Term Memory (LSTM) networks. Notably, eSPA achieved this at approximately 0.02% of the computational cost—a 5,000x increase in efficiency. Furthermore, because eSPA+ produces piecewise linear solutions on simplices, the model's "centers" have transparent physical meanings, allowing scientists to pinpoint the geographic regions driving predictions.

In-Materio Computing

Skyrmion-based reservoir computers demonstrated state-of-the-art performance in spoken digit classification, achieving 97.4% accuracy. This highlights the potential for physical materials to perform complex AI tasks with a fraction of the energy required by silicon-based architectures.

Adversarial Robustness

To address the vulnerability of AI, researchers introduced the Minimal Adversarial Distance (MAD) metric. This provides a mathematical proof of a classifier's stability, defining exactly how much an input can be perturbed before the classification fails. This rigor is essential for establishing safety margins in scientific AI.

Limitations and Future Directions

Despite these advances, several challenges remain. While the workshop established methods for measuring adversarial vulnerability, developing robust defenses within the entropic framework is an ongoing effort. Additionally, theoretical work on Sobolev and Besov spaces suggests that while DNNs can reach optimal approximation rates, these rates may be difficult to realize numerically with fixed-bit encoding.

Future research will focus on scaling eSPA+ to handle increasingly complex and heterogeneous datasets without sacrificing the interpretability and efficiency that make it ideal for the natural sciences. The goal remains the establishment of a unified mathematical foundation that supports both rigorous analysis and practical application in the "small data" era.

Summary updated

Abstract

The mathematical framework of approximate entropic learning introduced very recently promises to provide robust, cheap and efficient ways of machine learning in the so called “small data” regime, when the underlying learning task is highly-underdetermined, due to a large problem dimension and relatively small data statistics size. Such “small data” learning challenges are particularly common in the natural sciences (e.g., in geosciences, in climate research, in economics, and in biomedicine), imposing considerable difficulties for the numerics of common “data-hungry” Artificial Intelligence (AI) tools like Deep Learning (DL). The aim of this workshop will be to bring together experts in the emergent fields of entropic and DL mathematics/numerics, with some lead experts applying AI in the domain disciplines. The goal will be to detect and to discuss the commonalities in the challenges and in their mathematical solutions, as well as to discuss and fine-tune common mathematical problem formulations that are motivated by the AI applications in natural sciences. The establishment of a common mathematical framework for such small-data machine learning tasks would not only bolster future methodological developments but would also lay solid foundations to further in-depth rigorous analysis and theoretically founded interpretation of these methods and their results.
BibTeX
@inproceedings{gerber2024mini,
  author = {Susanne Gerber and Illia Horenko and Rupert Klein and T. Okane},
  title = {{Mini-Workshop: Mathematics of Entropic AI in the Natural Sciences}},
  booktitle = {Oberwolfach},
  year = {2024},
  doi = {10.4171/owr/2024/18},
}
Figure