Piecewise Linear Transformation - Propagating Aleatoric Uncertainty in Neural Networks

Thomas Krapf, Michael Hagn, Paul Miethaner, Alexander Schiller, Lucas Luttner, Bernd HeinrichAAAI 2024 · DOI

TL;DR

This paper introduces Piecewise Linear Transformation (PLT), a novel method for accurately propagating aleatoric uncertainty (data noise) through neural networks (NNs) with piecewise linear activation functions like ReLU. PLT offers a theoretically grounded and empirically validated way to quantify predictive uncertainty, crucial for high-stakes applications.

What Problem Does This Paper Address?

Neural networks are increasingly used in critical domains like autonomous driving and medical diagnostics. However, standard NNs provide only deterministic outputs, lacking a measure of confidence or uncertainty. This is problematic because real-world data is often noisy (aleatoric uncertainty), leading to potentially overconfident and unreliable predictions. While methods exist to propagate this uncertainty, they often suffer from significant approximation errors and struggle to accurately represent the true predictive uncertainty in the NN's output. This paper aims to bridge this gap by proposing a method that propagates aleatoric uncertainty precisely, especially for NNs with piecewise linear activation functions.

Methods

The core of the proposed method, Piecewise Linear Transformation (PLT), lies in leveraging the inherent piecewise linear structure of NNs with activation functions like ReLU. Instead of relying on sampling-based methods or making restrictive assumptions about probability distributions (PDs) (e.g., assuming they are always Gaussian), PLT directly analyzes the structure of the NN.

  1. Piecewise Linear Representation: The paper first establishes that NNs with piecewise linear activations can be viewed as piecewise linear functions. This means the function can be broken down into simpler, linear segments over different regions of the input space.
  2. Exact Propagation through Linear Regions: For each of these linear segments (regions), the paper derives an exact formula for propagating probability distributions. This is a key step, as it avoids approximations within these simpler parts of the network.
  3. Piecewise Probability Density Functions (PPDFs): To handle the complex, potentially non-standard PDs that can arise in NNs, the authors introduce the concept of Piecewise Probability Density Functions (PPDFs). This allows for a more general representation of uncertainty across the piecewise linear regions.
  4. Grid-Based Approximation: To make the exact formulas computationally tractable, especially for complex input PDs, PLT employs a grid-based approximation. The input space is finely subdivided, and the PD is approximated as a piecewise constant function on this grid. This allows for the propagation of uncertainty through the NN's linear regions.
  5. Output PPDF: The method then combines the propagated uncertainty from each linear region to form the final output PPDF. The paper also shows how this can be rendered into a piecewise constant form in the output space, akin to a multidimensional histogram, which can be made arbitrarily exact.

Datasets

The paper evaluates its method on a broad range of publicly available real-world datasets for both classification and regression tasks. Specific dataset names mentioned include: Appendicitis, Banana, Balance, Bands, Boston, Breast Cancer, California Housing, Diabetes, Iris, Machine, Real Estate, Vehicle, and Wine.

The authors note that these datasets do not inherently exhibit aleatoric uncertainty represented by PDs. Therefore, they induce uncertainty by selecting features with high importance and labeling their values as uncertain with a fixed probability (e.g., 50%). This induced uncertainty is then modeled using a Gaussian kernel density estimation on multiple imputed values, creating a continuous PD for the uncertain features.

Key Insights and Ideas

  • Leveraging NN Structure: The most significant insight is that the piecewise linear nature of common NNs isn't just an architectural detail but a powerful property that can be exploited for exact uncertainty propagation.
  • Beyond Gaussian Assumptions: Traditional methods often assume Gaussian distributions, which is a major limitation. PLT's ability to handle arbitrary PDs without such assumptions is a critical advantage.
  • Dependencies Matter: The paper emphasizes that ignoring dependencies between neurons within a layer leads to significant approximation errors. PLT preserves these dependencies by working with joint PDs.
  • Theoretical Exactness: The method is grounded in theoretical proofs that bound approximation errors and demonstrate that the propagated PD can be made arbitrarily exact by refining the grid.

Key Results

The paper presents both theoretical and empirical results:

  • Theoretical Exactness: The authors prove that PLT can achieve theoretically exact propagation of aleatoric uncertainty, with approximation errors converging to zero as the grid resolution increases.
  • Empirical Superiority: In experiments on various real-world datasets, PLT significantly outperformed competing methods (ADF, AS, UT) across multiple metrics (L1, L2, Hellinger distances). These metrics quantify the difference between the propagated PD and a high-quality ground truth generated via Monte Carlo simulation.
  • Error Reduction: Methods making restrictive assumptions (like ADF and UT) showed high errors, confirming the theoretical findings. The sample-based method (AS) also suffered from limitations in estimating complex PDs.

Limitations and Future Work

  • Trained NN Dependency: PLT requires an already trained NN. The accuracy of the propagated uncertainty is therefore dependent on the quality of the trained model.
  • Activation Function Restriction: The current PLT method is specifically designed for NNs with piecewise linear activation functions. While any activation function can be approximated by piecewise linear ones, this adds a layer of approximation.
  • Training Phase Uncertainty: PLT addresses uncertainty propagation after training. Integrating PLT into the training phase itself is an open area.
  • Architecture Extension: The authors suggest exploring the integration of PLT into different NN architectures, such as Convolutional Neural Networks (CNNs), as a promising future direction.

Summary updated

BibTeX
@inproceedings{krapf2024piecewise,
  author = {Thomas Krapf and Michael Hagn and Paul Miethaner and Alexander Schiller and Lucas Luttner and Bernd Heinrich},
  title = {{Piecewise Linear Transformation - Propagating Aleatoric Uncertainty in Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i18.30029},
}
Figure