Abstract
Modeling hybrid distributions—those involving both discrete and continuous variables—is challenging when the data must satisfy complex logical constraints. Traditional density estimators often fail to respect these boundaries, assigning probability mass to impossible regions. We present LARIAT (LeARning to IntegrATe), a framework that combines logic-based support learning with density estimation. By using Satisfiability Modulo Theories (SMT) to learn precise supports and normalizing candidate densities, LARIAT ensures valid probability distributions that respect real-world constraints. Experiments on synthetic and real-world datasets demonstrate superior support recovery and log-likelihood performance.
Introduction
In many real-world applications, data is "hybrid," containing both Boolean categories and real numbers. Furthermore, this data is often highly "structured" by physical laws or safety regulations that create strict boundaries on what configurations are possible. These boundaries define the support of the distribution.
Existing density estimators, such as Mixed Sum-Product Networks (MSPNs) or Density Estimation Trees (DETs), are effective at finding patterns but struggle with these boundaries. They typically assume "axis-aligned" supports—simple boxes around variables. Consequently, they fail to capture "oblique" constraints (e.g., $X_1 + X_2 \leq 10$), leading them to assign probability mass to "illegal" or impossible regions. In safety-critical AI, this represents a significant reliability failure.
The LARIAT Framework
The authors introduce LARIAT (LeARning to IntegrATe), which decouples the learning process into two distinct steps: support learning and density normalization.
1. Support Learning with INCAL+
LARIAT extends a logic-learning algorithm called INCAL to work with "positive-only" data. Since real-world observations typically only include valid scenarios, INCAL+ creates a virtual "negative space" by defining a distance threshold ($\theta$) around observed points. It then employs Satisfiability Modulo Theories (SMT) to find a logical formula—specifically SMT(LRA)—that wraps around the valid data points while excluding the synthesized negative space.
2. Density Estimation and Normalization
LARIAT utilizes standard tools like DETs or MSPNs to learn a candidate density function. However, because these tools ignore the complex support learned in the first step, a Normalization step is required. LARIAT "trims" the density function to fit the learned support and redistributes the probability mass so the total integral remains 1.
The framework supports two types of normalization: Global (uniform scaling) and Local (precise redistribution within sub-regions), ensuring the model remains computationally tractable while respecting non-axis-aligned boundaries.
Experimental Evaluation
The researchers evaluated LARIAT using synthetic data with varying logical constraints and 18 real-world hybrid benchmarks from the UCI/MLC++ library.
Key Results
- Superior Support Recovery: INCAL+ significantly outperformed "trivial" supports (simple bounding boxes) in recovering the true shape of synthetic distributions. As shown in the volume difference analysis, the logic-based approach maintains much higher fidelity to the true support.
- Improved Log-Likelihood: On real-world UCI datasets, LARIAT consistently achieved higher log-likelihoods than base estimators alone, often with statistical significance ($p < 0.0001$).
- Safety and Validity: Unlike standard estimators, LARIAT guarantees that the resulting model is a valid Weighted Model Integration (WMI) distribution, allocating zero mass to infeasible configurations.
Discussion and Future Work
LARIAT demonstrates that probability density should not be learned in a vacuum. By using formal logic as a "guardrail," the probabilistic model becomes more robust and reliable. However, normalization is a #P-hard problem, which can be computationally expensive for high-dimensional data. Future research will explore search-based program synthesis for support learning and testing LARIAT with even newer density estimators like Hybrid Sum-Product Networks (HSPNs).
Summary updated