Learning Weighted Model Integration Distributions

Paolo Morettin, Samuel Kolb, Stefano Teso, Andrea Passerini

Abstract

Modeling hybrid distributions—those involving both discrete and continuous variables—is challenging when the data must satisfy complex logical constraints. Traditional density estimators often fail to respect these boundaries, assigning probability mass to impossible regions. We present LARIAT (LeARning to IntegrATe), a framework that combines logic-based support learning with density estimation. By using Satisfiability Modulo Theories (SMT) to learn precise supports and normalizing candidate densities, LARIAT ensures valid probability distributions that respect real-world constraints. Experiments on synthetic and real-world datasets demonstrate superior support recovery and log-likelihood performance.

Introduction

In many real-world applications, data is "hybrid," containing both Boolean categories and real numbers. Furthermore, this data is often highly "structured" by physical laws or safety regulations that create strict boundaries on what configurations are possible. These boundaries define the support of the distribution.

Existing density estimators, such as Mixed Sum-Product Networks (MSPNs) or Density Estimation Trees (DETs), are effective at finding patterns but struggle with these boundaries. They typically assume "axis-aligned" supports—simple boxes around variables. Consequently, they fail to capture "oblique" constraints (e.g., $X_1 + X_2 \leq 10$), leading them to assign probability mass to "illegal" or impossible regions. In safety-critical AI, this represents a significant reliability failure.

The LARIAT Framework

The authors introduce LARIAT (LeARning to IntegrATe), which decouples the learning process into two distinct steps: support learning and density normalization.

1. Support Learning with INCAL+

LARIAT extends a logic-learning algorithm called INCAL to work with "positive-only" data. Since real-world observations typically only include valid scenarios, INCAL+ creates a virtual "negative space" by defining a distance threshold ($\theta$) around observed points. It then employs Satisfiability Modulo Theories (SMT) to find a logical formula—specifically SMT(LRA)—that wraps around the valid data points while excluding the synthesized negative space.

2. Density Estimation and Normalization

LARIAT utilizes standard tools like DETs or MSPNs to learn a candidate density function. However, because these tools ignore the complex support learned in the first step, a Normalization step is required. LARIAT "trims" the density function to fit the learned support and redistributes the probability mass so the total integral remains 1.

LARIAT combines a tree-based density estimator with logical constraints to model hybrid distributions.

The framework supports two types of normalization: Global (uniform scaling) and Local (precise redistribution within sub-regions), ensuring the model remains computationally tractable while respecting non-axis-aligned boundaries.

Experimental Evaluation

The researchers evaluated LARIAT using synthetic data with varying logical constraints and 18 real-world hybrid benchmarks from the UCI/MLC++ library.

Key Results

  • Superior Support Recovery: INCAL+ significantly outperformed "trivial" supports (simple bounding boxes) in recovering the true shape of synthetic distributions. As shown in the volume difference analysis, the logic-based approach maintains much higher fidelity to the true support.
  • Improved Log-Likelihood: On real-world UCI datasets, LARIAT consistently achieved higher log-likelihoods than base estimators alone, often with statistical significance ($p < 0.0001$).
  • Safety and Validity: Unlike standard estimators, LARIAT guarantees that the resulting model is a valid Weighted Model Integration (WMI) distribution, allocating zero mass to infeasible configurations.
Comparison of volume difference between true and learned supports, showing INCAL+ significantly outperforms trivial axis-aligned bounding boxes.

Discussion and Future Work

LARIAT demonstrates that probability density should not be learned in a vacuum. By using formal logic as a "guardrail," the probabilistic model becomes more robust and reliable. However, normalization is a #P-hard problem, which can be computationally expensive for high-dimensional data. Future research will explore search-based program synthesis for support learning and testing LARIAT with even newer density estimators like Hybrid Sum-Product Networks (HSPNs).

Summary updated

Abstract

Weighted model integration (WMI) is a framework for probabilistic inference over distributions with discrete and continuous variables and structured supports. Despite the growing popularity of WMI, existing density estimators ignore the problem of learning a structured support, and thus fail to handle unfeasible configurations and piecewise-linear relations between continuous variables. We propose lariat, a novel method to tackle this challenging problem. In a first step, our approach induces an SMT(ℒℛA) formula representing the support of the structured distribution. Next, it combines the latter with a density learned using a state-of-the-art estimation method. The overall model automatically accounts for the discontinuous nature of the underlying structured distribution. Our experimental results with synthetic and real-world data highlight the promise of the approach.
BibTeX
@inproceedings{morettin2020learning,
  author = {Paolo Morettin and Samuel Kolb and Stefano Teso and Andrea Passerini},
  title = {{Learning Weighted Model Integration Distributions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5967},
}
Figure