Integrating Probabilistic Rules into Neural Networks: A Stochastic EM Learning Algorithm

Gerhard PaassUAI 1991

TL;DR

This paper introduces a novel stochastic EM algorithm to integrate probabilistic rules (expert knowledge) with associative neural networks (data-driven learning). This allows for a more robust way to model complex systems where both types of information are available but potentially conflicting, enabling the handling of uncertain evidence and even cyclic dependencies.

What Problem Does This Paper Address?

Modeling complex systems often involves two types of information: explicit rules derived from expert knowledge or theoretical understanding, and associative relationships learned from data. Probabilistic inference networks excel at representing the former, while associative neural networks (like Boltzmann machines) are good at capturing the latter, especially when dealing with hidden variables. However, these two approaches often produce conflicting information, and integrating them into a single, coherent model has been a challenge. This paper addresses the problem of how to combine these two sources of information in a principled way, creating a unified probabilistic model that can handle uncertainty and learn from potentially incomplete or contradictory data.

Methods

The core contribution is a stochastic EM (Expectation-Maximization) algorithm. The EM algorithm is a standard technique for finding maximum likelihood estimates when data is missing. This paper adapts it to a "stochastic" version, meaning it uses random sampling (stochastic simulation) to handle the missing data.

Here's a breakdown of the approach:

  • Probabilistic Networks: The paper starts by defining probabilistic inference networks, which use rules to describe relationships between variables. These rules can be expressed as constraints on probability distributions.
  • Associative Neural Networks: It then introduces associative neural networks (specifically mentioning Boltzmann machines) that use weighted links to model correlations between variables, often employing "hidden" variables to capture complex dependencies.
  • Integration: The key idea is to combine these two by defining a joint probability distribution that incorporates both the probabilistic rules and the associative structure. This is achieved by representing both types of information as terms in a generalized probability distribution function (equation 4).
  • Stochastic EM Algorithm:
    • E-step (Expectation): This step involves "imputing" missing data. Instead of deterministically filling in missing values, it randomly generates them based on the current estimated parameters of the network and the observed data. This is done using stochastic simulation, leveraging the conditional probabilities derived from the network structure. For truncated samples (where some data is intentionally removed), it estimates the expected size of the missing portion.
  • M-step (Maximization): Once the missing data is imputed (even if stochastically), the algorithm performs a standard maximum likelihood estimation to update the network's parameters (represented by λ). This step uses the imputed data as if it were fully observed.
  • Convergence: The algorithm iterates between the E and M steps until the parameters converge to a stationary point, ideally a local minimum of the likelihood function. The stochastic nature of the E-step is hypothesized to help escape local minima, a common problem in EM algorithms.

Datasets

Not discussed in the paper. The paper focuses on a general framework and uses hypothetical examples and data structures (like Table 1) to illustrate its concepts rather than specific, named datasets.

Key Insights and Ideas

  • Unified Framework: The paper successfully bridges the gap between rule-based probabilistic reasoning and data-driven associative learning within a single probabilistic framework. This is a significant step towards building more comprehensive and flexible AI models.
  • Handling Uncertainty and Contradiction: By using a probabilistic approach and the EM algorithm, the method can naturally handle uncertain evidence and even contradictory information from different sources, finding a compromise based on their relative reliability.
  • Stochasticity for Robustness: The introduction of a stochastic E-step is a clever way to address the missing data problem. It allows the algorithm to explore the parameter space more broadly and potentially avoid getting stuck in suboptimal solutions (local minima) that deterministic imputation might lead to.
  • Flexibility in Network Structure: The approach doesn't require acyclic network structures and can handle hidden, unobservable variables, making it applicable to a wider range of real-world problems.

Key Results

The paper presents a theoretical framework and an algorithm. It does not report quantitative results from experiments on specific datasets. However, it highlights the potential for the algorithm to:

  • Integrate probabilistic rules and associative data in a sound way.
  • Handle cycles of probabilistic rules and hidden variables.
  • Process uncertain and contradictory evidence.
  • The stochastic EM algorithm is presented as an alternative to traditional EM for missing data problems, with empirical evidence suggesting it can escape local minima.

Limitations and Future Work

  • Computational Intensity: The paper acknowledges that Boltzmann machines, which are related to the associative part of the network, are computationally intensive. While the stochastic EM aims to be an alternative, empirical investigations are ongoing to determine its relative computational efficiency compared to other methods.
  • Approximation of Maximum Entropy: The paper notes that when probabilistic rules are "softly" integrated (as data), the resulting network is an approximation to the maximum entropy distribution where rules are "hardwired." The conditional independence assumptions implied by the rule network structure might not always be perfectly valid, and further research is needed to assess the impact of such errors.
  • Future Work: The authors suggest that further research is needed to judge the effect of approximation errors and to compare the "soft" integration approach with a "hardwired" approach. They also mention ongoing empirical investigations into the computational efficiency of the stochastic EM approach.

Summary updated

BibTeX
@inproceedings{paass1991integrating,
  author = {Gerhard Paass},
  title = {{Integrating Probabilistic Rules into Neural Networks: A Stochastic EM Learning Algorithm}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {1991},
}
Figure