Maximum Probability Theorem: A Framework for Probabilistic Machine Learning

Amir Emad Marvasti, Ehsan Emad Marvasti, Ulas Bagci, Hassan ForooshIEEE Trans. Artif. Intell 2021 · OpenReview

TL;DR

This paper introduces the Maximum Probability (MP) framework for probabilistic machine learning, redefining models as events in a probability space and quantifying their probabilities using a novel MP theorem. This approach automates regularization by maximizing model probabilities, offering a more principled and flexible alternative to traditional Bayesian methods, especially for complex models.

What Problem Does This Paper Address?

Traditional probabilistic machine learning, particularly within the Bayesian framework, relies heavily on choosing prior distributions over model parameters. This choice significantly impacts regularization and model learning. However, for complex models, determining appropriate priors can be intractable, often requiring ad-hoc justifications or experimental tuning. The paper addresses this challenge by proposing a new framework that shifts the focus from parameter priors to the probability of the model itself as an event, thereby automating and unifying the regularization process.

Methods

The core of the paper is the Maximum Probability (MP) Theorem. This theorem establishes an upper bound on the probability of an event (representing a model) based on its conditional distribution with respect to an arbitrary random variable. This allows for the quantification of a model's probability without explicitly defining priors on its parameters.

The MP framework then defines a model as an event in the probability space, and the learning objective becomes maximizing the probability of set operations between the model event and the "oracle" event (representing the true underlying model). Two key objective functions are explored:

  • Log-Likelihood Objective: This is a standard approach, but the MP framework provides a new perspective on its regularization properties.
  • Intersection Objective: This function maximizes the probability of the intersection between the model event and the oracle event. This is shown to inherently provide regularization, preventing degenerate solutions and overfitting.

The framework also introduces the concept of a-skeletons, which are derived from conditional distributions and help in understanding the behavior of the objective functions, particularly how they concentrate probability mass.

Datasets

The paper uses the CIFAR10 image dataset for evaluating the proposed framework in the context of Convolutional Neural Networks (CNNs). The dataset consists of 10 classes with 50,000 training samples and 10,000 test samples.

Key Insights and Ideas

  • Models as Events: Redefining models as events in a probability space provides a more fundamental way to reason about their probabilities and relationships.
  • Automated Regularization: The MP theorem allows for the direct maximization of model probability, which inherently acts as a regularizer, eliminating the need for manual prior selection.
  • Prior over Observables: The MP framework requires a prior over the observable random variable, which is often more readily available or intuitive than priors over model parameters.
  • Unified Framework: The MP theorem provides a theoretical foundation that connects various existing concepts in probabilistic learning and information theory.
  • Intersection Objective as a Regularizer: Maximizing the intersection probability between the model and the oracle naturally leads to regularization, preventing overfitting and leading to more stable solutions.

Key Results

The paper demonstrates the effectiveness of the MP framework through theoretical derivations and experimental results on a Bernoulli random variable and CNNs.

  • Bernoulli Example: Visualizations show that the intersection objective function, unlike the log-likelihood, provides a unique and finite solution for critical points, especially for $\alpha > 1$. As $\alpha$ approaches infinity, the solution tends towards a trivial solution, but for finite $\alpha$, it offers a well-behaved objective.
  • CNN Application: Experiments on CIFAR10 show that the intersection objective function, when implemented with a "HyperNormalization" layer, acts as a regularizer. The results in Table I indicate that increasing the $\alpha$ parameter in the intersection objective function leads to a smaller gap between training and test loss, suggesting better generalization compared to standard L2 regularization, especially in terms of loss. While test accuracy didn't always improve significantly, the test loss convergence was more stable.

Limitations and Future Work

  • Finite-Range Variables: The current framework is primarily developed for finite-range random variables. Generalizing it to continuous random variables is identified as a key area for future work.
  • Prior Selection: While the framework simplifies the problem by requiring priors only over observables, the selection of these priors still relies on existing principles like MAXENT or Laplace's Principle.
  • Network Depth: For deep networks like CNNs, the regularization effect of the intersection objective might diminish due to gradient vanishing. The authors hypothesize that incorporating more random variables in intermediate layers could improve test accuracy.
  • Exploring Other Set Operations: The paper suggests exploring other set operations (e.g., symmetric difference) as objective functions to avoid trivial solutions.

Summary updated

Abstract

We present a theoretical framework of probabilistic learning derived from the maximum probability (MP) theorem shown in this article. In this probabilistic framework, a model is defined as an event in the probability space, and a model or the associated event —either the true underlying model or the parameterized model—has a quantified probability measure. This quantification of a model's probability measure is derived by the MP theorem , in which we have shown that an event's probability measure has an upper bound given its conditional distribution on an arbitrary random variable. Through this alternative framework, the notion of model parameters is encompassed in the definition of the model or the associated event . Therefore, this framework deviates from the conventional approach of assuming a prior on the model parameters. Instead, the regularizing effects of assuming prior over parameters are imposed through maximizing probabilities of models or, according to information theory, minimizing the information content of a model. The probability of a model in our framework is invariant to reparameterization and is solely dependent on the model's likelihood function. In addition, rather than maximizing the posterior in a conventional Bayesian setting, the objective function in our alternative framework is defined as the probability of set operations (e.g., intersection) on the event of the true underlying model and the event of the model at hand. Our theoretical framework adds clarity to probabilistic learning through solidifying the definition of probabilistic models, quantifying their probabilities, and providing a visual understanding of objective functions. Impact Statement—The choice of prior distribution over the parameters of probabilistic machine learning models determines the regularization of learning algorithms in the Bayesian perspective. The complexity in choice of prior over the parameters and the form of regularization is relative to the complexity of the models being used. Thereby, finding priors for parameters of complex models is often not tractable. We address this problem by uncovering the maximum probability (MP) theorem as a direct consequence of Kolmogorov's probability theory. Through the lens of the MP theorem, the process of regularizing models is understood and automated. The regularization process is defined as the maximization of the probability of the model. The probability of the model is understood by the MP theorem and is determined by the behavior of the model. The effects of maximizing the probability of the model can be backpropagated in a gradient-based optimization process. Consequently, the MP framework provides a form of black-box regularization and eliminates the need for case-by-case analysis of models to determine priors.
BibTeX
@article{marvasti2021maximum,
  author = {Amir Emad Marvasti and Ehsan Emad Marvasti and Ulas Bagci and Hassan Foroosh},
  title = {{Maximum Probability Theorem: A Framework for Probabilistic Machine Learning}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2021},
  url = {https://openreview.net/forum?id=pm9vOLQC18},
}
Figure