An entropy-optimal path to humble AI

Davide Bassetti, Lukás Pospísil, Michael Groom, Terence J. O'Kane, Illia HorenkoarXiv 2025 · DOI · arXiv

Abstract

This paper introduces the Entropy-Optimal feed-forward Network (EON), a novel mathematical framework that reformulates Boltzmann Machines to create computationally efficient and mathematically transparent AI. By replacing traditional gradient descent with analytical coordinate descent, EON achieves state-of-the-art performance on complex climate and biomedical tasks using a fraction of the data and parameters required by standard deep learning models. Crucially, EON provides explicit reliability measures, allowing models to identify out-of-distribution inputs and avoid over-confident "hallucinations."

Introduction

Modern deep learning, while powerful, faces significant challenges: astronomical computational costs, optimization fragility due to "black-box" hyperparameters, and a lack of robustness. State-of-the-art models are often over-confident, providing answers even when input data is far outside their training experience. To address these issues, the authors propose a "humble" AI framework that prioritizes mathematical transparency and resource efficiency.

Methodology: The EON Framework

The Entropy-Optimal feed-forward Network (EON) moves away from standard perceptron architectures, instead utilizing a non-equilibrium entropy-optimizing reformulation of Boltzmann Machines.

Gradient-Descent-Free Learning

A core innovation of EON is its departure from backpropagation. Instead, it employs a coordinate descent algorithm. Because the framework is built on the exact law of total probability and convex polytope representations, the sub-problems for each layer possess closed-form analytical solutions. This eliminates the need for tuning learning rates or batch sizes and provides guarantees regarding the optimization process.

Iterative coordinate descent algorithm for EON parameter optimization.

Key Technical Features

  • Entropy Regularization: EON uses Shannon entropy constraints to navigate between deterministic "hard" assignments and uniform "soft" distributions, ensuring balanced learning.
  • Input Reliability Measure ($\gamma_{(0)}$): Unlike standard neural networks, EON includes an input layer that measures how well a new data point fits within the "convex hull" of the training data. This allows the model to explicitly flag low confidence for "weird" or adversarial inputs.
  • Bilinear Forms: By using bilinear forms for energy instead of the quadratic forms found in classic Boltzmann Machines, EON simplifies the underlying mathematics and allows the application of Jensen’s inequality to find optimal upper bounds for the loss function.

Experimental Validation

The researchers validated EON across a diverse battery of tests, including:

  • Synthetic Benchmarks: Regression and classification tasks designed to compare model complexity against theoretical limits.
  • Real-World Medical Data: 13 datasets covering diagnostics for diabetes, lung cancer, heart failure, and leukemia.
  • Climate Science: Historical climate records (1975–2007) used to predict El Niño and La Niña (ENSO) phases based on sea surface and deep ocean temperature gradients.

Results and Insights

Approaching Kolmogorov Complexity

EON models are remarkably "slim." Their descriptor length (the number of parameters) is very close to the Kolmogorov Complexity—the theoretical minimum information needed to represent a problem. In the "lung auscultation" task, for instance, EON required only 13 tunable parameters, while the best standard model required over 4,000.

Performance and Efficiency

  • Climate Prediction: EON demonstrated systematically higher prediction skills for ENSO onsets compared to standard neural networks and ensemble models like Random Forest.
  • Data Efficiency: EON reached maximum performance plateaus using only 10-18 years of climate data, whereas traditional tools often require decades of data.
  • Accuracy: EON achieved the lowest Root Mean Square Error (RMSE) across all six regression benchmarks and the highest Area Under the Curve (AUC) in five out of seven classification benchmarks.

Conclusion and Future Directions

EON represents a significant step toward more reliable and efficient AI, particularly for structured data in critical fields like medicine and climate science. While currently focused on supervised learning, future work will explore extending the EON framework to generative modeling, reinforcement learning, and the integration of multi-modal data sources.

Summary updated

BibTeX
@misc{bassetti2025entropy,
  author = {Davide Bassetti and Lukás Pospísil and Michael Groom and Terence J. O'Kane and Illia Horenko},
  title = {{An entropy-optimal path to humble AI}},
  year = {2025},
  eprint = {2506.17940},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.17940},
}
Figure