Estimating Continuous Distributions in Bayesian Classifiers

George H. John, Pat LangleyUAI 1995

TL;DR

This paper introduces "Flexible Bayes," a novel approach to Bayesian classification that replaces the traditional assumption of normally distributed continuous features with nonparametric kernel density estimation. This allows for more accurate modeling of real-world data, leading to significant improvements in classification accuracy across various domains.

What Problem Does This Paper Address?

Traditional Bayesian classifiers, particularly the "naive Bayes" variant, often make a simplifying assumption: continuous features are distributed according to a single Gaussian (normal) distribution within each class. While this assumption makes calculations tractable and learning efficient, it can lead to suboptimal performance when the true data distribution deviates significantly from a Gaussian. This is a common scenario in real-world datasets, where feature distributions can be complex, multimodal, or otherwise non-normal. The paper aims to bridge this gap by developing a Bayesian classifier that can handle these more complex continuous distributions without sacrificing the core benefits of the Bayesian approach.

Methods

The core innovation is the replacement of the Gaussian assumption with kernel density estimation (KDE) for continuous attributes.

  • Naive Bayes (Standard): Models each continuous attribute's distribution within a class using a single Gaussian, characterized by its mean and standard deviation. These parameters are estimated using maximum likelihood (sample mean and standard deviation).
  • Flexible Bayes (Novel): Employs kernel density estimation. For each continuous attribute and class, it estimates the probability density by averaging a set of Gaussian kernels centered at each observed data point for that attribute and class. This is mathematically represented as: $p(X = x|C = c) = \sum_{i} g(x, \mu_i, \sigma_i)$ where $g$ is a Gaussian kernel, $\mu_i$ and $\sigma_i$ are the mean and standard deviation of the $i$-th training instance for attribute $X$ in class $C$, and the sum is over all training instances $i$ belonging to class $c$. The kernel width $\sigma$ is set to $1/\sqrt{n_c}$, where $n_c$ is the number of training instances in class $c$, making the estimate more local as more data becomes available.

The paper also rigorously proves the strong pointwise consistency of Flexible Bayes, demonstrating that its estimates converge to the true density functions as the amount of training data increases. This theoretical foundation assures that, in the limit, Flexible Bayes will achieve the Bayes optimal error rate.

Datasets

The paper evaluates its approach on 11 natural datasets from the UCI machine learning repository. These datasets were chosen because they contained a significant fraction of numeric features, making them suitable for contrasting the performance of Gaussian-based and kernel density estimation-based classifiers. The paper does not introduce new benchmark datasets but uses existing ones to demonstrate the practical utility of Flexible Bayes.

Key Insights and Ideas

  • Normality is not always best: The paper strongly suggests that the Gaussian assumption, while convenient, is often too restrictive for real-world data, leading to performance degradation.
  • Kernel density estimation is a powerful tool for Bayesian models: By adopting KDE, Flexible Bayes can capture more complex density shapes, leading to improved classification accuracy.
  • Theoretical guarantees matter: The paper provides strong theoretical backing (strong pointwise consistency) for Flexible Bayes, assuring its asymptotic optimality.
  • Trade-offs exist: While Flexible Bayes offers improved accuracy, it comes with increased computational and storage complexity compared to Naive Bayes, as it needs to store all continuous attribute values for kernel estimation.
  • Data characteristics influence performance: The paper observes that in some medical domains, where features might be intentionally defined to be roughly normal, Naive Bayes can perform comparably or even better than Flexible Bayes. This highlights the importance of understanding the underlying data distributions.
  • Data quirks can confuse flexible models: The study found that systematic rounding errors in blood pressure measurements (e.g., to the nearest 10) confused the kernel density estimator, leading to worse performance than Naive Bayes on that specific attribute. This suggests that preprocessing or handling such data artifacts can be crucial.

Key Results

  • Significant accuracy improvements: Flexible Bayes achieved significantly higher accuracy than Naive Bayes in 5 out of 11 natural domains.
  • Competitive performance: In several other domains, Flexible Bayes performed comparably to Naive Bayes, and in some cases, it outperformed the well-known C4.5 decision tree algorithm.
  • Hypothesis validation:
    • When normality holds, Naive Bayes reaches its asymptotic performance faster with smaller training sets, while Flexible Bayes catches up as the dataset size increases.
  • When normality is violated, Flexible Bayes approaches the Bayes optimal error rate, while Naive Bayes struggles due to its overly simplistic model.
  • Complexity increase: Flexible Bayes has higher space complexity (storing all continuous attribute values) and time complexity for testing (O(mnk) vs. O(mk) for Naive Bayes, where m is test cases, n is training cases, k is features).

Limitations and Future Work

  • Computational Cost: The primary limitation is the increased computational and storage cost of kernel density estimation, especially for large datasets.
  • Kernel Width Selection: The paper uses a simple heuristic for setting the kernel width ($\sigma = 1/\sqrt{n_c}$). The authors acknowledge that more sophisticated methods for adaptive kernel width selection (e.g., using cross-validation) could further improve performance.
  • Handling Data Artifacts: The study identified issues with systematic rounding in data (e.g., blood pressure measurements) that can negatively impact kernel density estimation. Future work could explore methods to detect and mitigate such artifacts.
  • Generalization to Other Bayesian Models: The paper focuses on the naive Bayes classifier. Extending the kernel density estimation approach to more complex Bayesian networks with arbitrary dependencies is a promising avenue.
  • Bayesian Estimation of Kernels: Exploring Bayesian approaches to density estimation with Gaussian mixtures, similar to Cheeseman et al. (1988), is suggested.

Summary updated

BibTeX
@inproceedings{john1995estimating,
  author = {George H. John and Pat Langley},
  title = {{Estimating Continuous Distributions in Bayesian Classifiers}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {1995},
}
Figure