A Distribution Similarity Based Regularizer for Learning Bayesian Networks

Weirui Kong, Wenyi WangarXiv 2018 · OpenReview

TL;DR

This paper introduces a novel "distribution similarity based regularizer" for learning Bayesian networks, specifically applied to modeling wave propagation in inhomogeneous media. By encouraging similar conditional probability distributions between neighboring nodes, their method effectively learns complex dynamics that simpler models fail to capture, demonstrating its practical value in material damage identification.

What Problem Does This Paper Address?

Modeling complex physical phenomena like wave propagation in inhomogeneous materials is challenging because closed-form solutions are often unavailable, and the underlying physics can be intricate. Traditional approaches often rely on strong assumptions or extensive feature engineering. The authors aim to learn these dynamics directly from data using Bayesian networks, but face the problem of limited data leading to overfitting and poor generalization. They propose a regularization technique to inject prior knowledge about the expected similarities between related parts of the system, specifically focusing on the conditional probability distributions of neighboring nodes.

Methods

The core contribution is a "distribution similarity based penalization" approach. Instead of directly forcing the parameters of neural networks representing conditional probability distributions to be similar (parameter sharing), this method penalizes the statistical distance between the distributions themselves. They use KL divergence and Bhattacharyya distance as measures of dissimilarity between learned Gaussian conditional probability distributions. This regularization is added to the standard negative log-likelihood loss function. They compare this approach against three baselines: a "free" model (no regularization), strict parameter sharing, and multi-task learning (MTL).

Datasets

The authors generated a synthetic dataset simulating 2D elastic wave propagation (P-SV wave) using a finite-difference method on a plate with varying densities. This dataset consists of time sequences of wave function evaluations on a 50x50 grid. They used this simulated data for both training and testing their models.

Key Insights and Ideas

  • Distribution-level similarity is more effective than parameter-level: The paper argues that encouraging similar distributions is a more nuanced and effective way to inject prior knowledge than simply sharing network weights. This allows for more flexibility while still leveraging commonalities.
  • Inhomogeneous media requires careful regularization: Simple models like strict parameter sharing fail because they oversimplify the problem by assuming homogeneity. The proposed method, by focusing on local similarities, can better handle spatial variations.
  • The choice of parent nodes for regularization matters: They found that using the same set of parent nodes for evaluating the regularization term could bias the results. Randomly selecting parent nodes or using specific neighborhood values at the previous time step yielded different outcomes, with the latter showing better convergence.

Key Results

The proposed distribution similarity based penalization method, using both KL divergence and Bhattacharyya distance, significantly outperformed the baseline models (free model, strict parameter sharing, and MTL) on the wave propagation modeling task. The regularized models showed much faster convergence and better generalization, with the test loss consistently decreasing. The "free" model exhibited unstable performance, while strict parameter sharing and MTL models struggled to capture the underlying dynamics.

Limitations and Future Work

The authors acknowledge that their current work uses a synthetic dataset generated by a finite-difference method. In practice, real-world data might be sparser, with only a limited number of wave function locations available. Future work could explore learning transition models with unobserved data. They also note that they did not implement a parameter similarity based penalization, which would have been a more direct comparison to their distribution similarity approach, but believe their current results sufficiently demonstrate the value of their method. The proposed approach is also presented as generalizable to other systems solvable by finite-difference methods.

Summary updated

Abstract

Probabilistic graphical models compactly represent joint distributions by decomposing them into factors over subsets of random variables. In Bayesian networks, the factors are conditional probability distributions. For many problems, common information exists among those factors. Adding similarity restrictions can be viewed as imposing prior knowledge for model regularization. With proper restrictions, learned models usually generalize better. In this work, we study methods that exploit such high-level similarities to regularize the learning process and apply them to the task of modeling the wave propagation in inhomogeneous media. We propose a novel distribution-based penalization approach that encourages similar conditional probability distribution rather than force the parameters to be similar explicitly. We show in experiment that our proposed algorithm solves the modeling wave propagation problem, which other baseline methods are not able to solve.
BibTeX
@misc{kong2018distribution,
  author = {Weirui Kong and Wenyi Wang},
  title = {{A Distribution Similarity Based Regularizer for Learning Bayesian Networks}},
  year = {2018},
  eprint = {1808.06347},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.06347},
}

Loading related content...

Figure