On The Connection of Benford's Law and Neural Networks

Surya Kant Sahu, Abhinav Java, Arshad ShaikharXiv 2021 · OpenReview

Abstract

Benford's Law (BL), or the Significant Digit Law, defines the probability distribution of the first digit of numerical values in a data sample. This Law is observed in many naturally occurring datasets and can be seen as a measure of "naturalness," finding applications in anomaly and fraud detection. In this work, we investigate whether the distribution of neural network parameters relates to generalization capability. We define a metric, Model Enthalpy (MLH), which measures the closeness of a set of numbers to Benford's Law. Empirically, we show that MLH is a strong predictor of validation accuracy. Furthermore, we utilize MLH as an alternative to validation accuracy for early stopping, effectively removing the need for a held-out validation set. Our results indicate that using MLH for early stopping can even surpass traditional methods by allowing the model to train on 100% of available data. Finally, we explore the connection between BL and the Free Energy Principle, framing optimization as a process of minimizing total system energy to reach equilibrium.

Introduction

In contemporary deep learning, the standard practice involves partitioning data into training and validation sets. The validation set serves as a proxy for real-world performance, guiding hyperparameter tuning and early stopping to prevent overfitting. However, this "data tax" limits the volume of information available for the model's primary learning phase. This paper explores an alternative: identifying intrinsic properties within the network's weights themselves that signal generalization potential.

The researchers turn to Benford’s Law (BL), a mathematical phenomenon where the leading digit '1' occurs with a frequency of approximately 30.1%, while '9' occurs less than 5% of the time. By measuring how "Benford-like" a model's weights are, we can potentially gauge its "naturalness" and, by extension, its ability to generalize to unseen data.

Methodology: Model Enthalpy (MLH)

To quantify a network's adherence to Benford's Law, the authors introduce Model Enthalpy (MLH). MLH is defined as the Pearson Correlation Coefficient between the empirical distribution of the first digits of a network's weights and the ideal Benford distribution.

Comparison of first-digit distributions in trained and random networks against the theoretical Benford's Law curve.

As shown in Figure 1, trained models exhibit a first-digit distribution that closely mirrors Benford's Law, whereas random initializations deviate significantly. This suggests that the optimization process "naturalizes" the weight distribution.

Beyond MLH, the paper proposes the Enthalpy Information Criterion (EIC). Similar to AIC or BIC, EIC combines training accuracy with MLH to estimate the model's internal energy and generalization risk. This allows for MLH-based Early Stopping, where training is halted when the "Benford-ness" of the weights peaks, rather than relying on a validation set.

Key Results and Discussion

The effectiveness of MLH was validated across diverse architectures (CNNs, LSTMs, Transformers) and datasets (CIFAR10, Stanford Dogs, etc.).

Correlation with Generalization

A primary finding is the strong monotonic relationship between MLH and validation accuracy. MLH acts as a "canary in the coal mine"; when the model begins to overfit and validation accuracy plateaus or drops, MLH typically follows a similar trajectory.

Temporal tracking of training accuracy, validation accuracy, and Model Enthalpy (MLH) across 6,000 training steps.

Figure 3 illustrates this synchronization. The peak of the MLH curve aligns closely with the peak of the validation accuracy curve. Quantitatively, MLH shows a Spearman’s Rank Correlation of 0.583 with validation accuracy. When integrated with training accuracy via a Gaussian Process Regressor, this correlation increases to 0.774.

Superior Early Stopping

By using MLH for early stopping, researchers can utilize 100% of the dataset for training. In experiments on CIFAR10, this approach achieved a mean test accuracy of 58.97%, outperforming the standard validation-split approach (58.04%). This demonstrates that the benefit of training on more data outweighs the slight noise in the MLH signal compared to a true validation set.

Thermodynamic Interpretation

The paper draws a compelling analogy between neural network training and statistical mechanics. The authors suggest that optimization is akin to a thermodynamic process where MLH represents "heat released" and training accuracy represents "heat given." The learning system seeks to minimize its total internal energy to reach a state of equilibrium, which corresponds to a well-generalized model.

Conclusion and Future Work

This research establishes a robust link between the first-digit distribution of model parameters and their generalization power. By looking at the "fingerprints" of digits within a model, we can perform efficient early stopping and maximize data utility. Future work will investigate the applicability of MLH to Large Language Models (LLMs) and seek a deeper theoretical explanation for why Benford's Law emerges so consistently in optimized neural systems.

Summary updated

Abstract

Benford's Law (BL) or the Significant Digit Law defines the probability distribution of the first digit of numerical values in a data sample. This Law is observed in many naturally occurring datasets. It can be seen as a measure of naturalness of a given distribution and finds its application in areas like anomaly and fraud detection. In this work, we address the following question: Is the distribution of the Neural Network parameters related to the network's generalization capability? To that end, we first define a metric, MLH (Model Enthalpy), that measures the closeness of a set of numbers to Benford's Law and we show empirically that it is a strong predictor of Validation Accuracy. Second, we use MLH as an alternative to Validation Accuracy for Early Stopping, removing the need for a Validation set. We provide experimental evidence that even if the optimal size of the validation set is known before-hand, the peak test accuracy attained is lower than not using a validation set at all. Finally, we investigate the connection of BL to Free Energy Principle and First Law of Thermodynamics, showing that MLH is a component of the internal energy of the learning system and optimization as an analogy to minimizing the total energy to attain equilibrium.
BibTeX
@misc{sahu2021connection,
  author = {Surya Kant Sahu and Abhinav Java and Arshad Shaikh},
  title = {{On The Connection of Benford's Law and Neural Networks}},
  year = {2021},
  eprint = {2102.03313},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.03313},
}
Figure