Recognition Networks for Approximate Inference in BN20 Networks

Quaid MorrisUAI 2001

TL;DR

This paper introduces "recognition networks," a novel approach to approximate inference in large Bayesian networks. By training a neural network to directly predict posterior probabilities from observed evidence, these networks offer a fast and potentially accurate alternative to traditional methods, especially when dealing with complex real-world scenarios like medical diagnosis where not all information is available.

What Problem Does This Paper Address?

In many real-world applications, such as medical diagnosis, we rely on Bayesian networks (BNs) to model complex systems. However, performing exact inference (calculating probabilities of unobserved variables given evidence) in large BNs with loops is computationally intractable. Approximate inference methods are necessary, but existing approaches like variational methods and Monte Carlo methods have their own limitations: variational methods can be inaccurate, and Monte Carlo methods can require a prohibitive number of samples.

A significant challenge in these applications is that not all potential evidence is observed. The process of selecting which evidence to collect is often complex and introduces a "selection bias." For instance, in medical diagnosis, the choice of tests and questions depends on previous findings. Ignoring this bias can lead to inaccurate posterior probabilities. This paper tackles the problem of approximate inference in BNs, specifically addressing the impact of this observation bias and proposing a new method to handle it.

Methods

The core contribution is the introduction of recognition networks. These are essentially multilayer perceptrons (MLPs) trained to predict the posterior marginals of unobserved variables given observed evidence. The input to the MLP is a vector representing the states of the evidential nodes, and the output units predict the posterior probabilities of the corresponding unobserved variables.

The recognition network is trained using samples generated from the Bayesian network. Crucially, the training process incorporates the effect of the observation bias by using an augmented BN that models this bias. The network parameters are optimized once to learn a deterministic mapping from evidence to approximate posteriors. This allows for single feedforward passes for inference, making it very fast. The paper explores two parameterizations for the recognition network: logistic regression (LR) networks and MLPs.

Datasets

The paper uses a large binary, two-layer, noisy-OR (BN2O) network, designed to be similar in structure and complexity to the Quick Medical Reference (QMR-DT) network, a well-known benchmark for medical diagnosis. This BN2O network contains over 4000 potentially observable nodes (findings) and over 600 unobservable nodes (diseases).

The authors create their own benchmark test sets by forward sampling from an augmented BN2O network. These benchmarks are designed to mimic realistic diagnostic problems and allow for controlled variation of the observation bias. They generate eight benchmark test sets by combining different values of $p^+$ (proportion of positive findings unobserved) and $p^-$ (proportion of negative findings unobserved).

Key Insights and Ideas

  • Observation Bias Matters: The paper strongly emphasizes that ignoring or incorrectly modeling the observation bias (the process by which evidence is selected) can significantly impact the accuracy of approximate inference, even with a simple bias.
  • Recognition Networks as a Fast Alternative: Recognition networks offer a promising approach for fast approximate inference. By learning a direct mapping from evidence to posteriors, they bypass the iterative or sampling-intensive nature of many existing methods.
  • MLPs as Powerful Approximators: The inclusion of a hidden layer in the recognition network (MLP) allows it to act as a universal approximator, potentially achieving higher accuracy than single-layer LR networks.
  • Robustness to Bias Modeling: The experiments suggest that recognition networks perform well even when the assumed observation bias in training differs from the actual bias in the test set, indicating a degree of robustness.

Key Results

The paper compares recognition networks (both LR and MLP versions) against two established approximate inference methods: JJ99 (a variational method) and AIS-BN (a Monte Carlo method).

  • Recognition Networks Perform Well: Both LR and MLP recognition networks generally perform very well across various test sets, often achieving good cumulative ratio curves, especially for low ranks (meaning they place good diagnoses high in their D-lists).
  • MLP Outperforms LR: The MLP recognition network generally performs best, especially when the assumed observation bias matches the actual bias.
  • AIS-BN Strengths: AIS-BN shows particular suitability for approximate inference when only a few positive findings are available.
  • Impact of Bias: The results highlight that the choice of observation bias parameters ($p^+$ and $p^-$) influences the performance of inference algorithms, demonstrating the importance of modeling this bias.

Limitations and Future Work

  • Simplistic Bias Model: The paper acknowledges that their model of observation bias is basic. Real-world selection processes are more complex.
  • Negative Evidence: The current implementation of recognition networks does not distinguish between negative and unobserved findings. The authors suggest future work could incorporate negative evidence using 1-of-N coding for input units.
  • Generalization: Extending recognition networks to arbitrary Bayesian networks, potentially requiring 1-of-N coding for both input and output units, is a suggested future direction.
  • Computational Cost: While inference is fast, training recognition networks can still be computationally intensive.

Summary updated

BibTeX
@inproceedings{morris2001recognition,
  author = {Quaid Morris},
  title = {{Recognition Networks for Approximate Inference in BN20 Networks}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2001},
}
Figure