Modelling heterogeneous distributions with an Uncountable Mixture of Asymmetric Laplacians

Axel Brando, José A. Rodríguez-Serrano, Jordi Vitrià, Alberto Rubio

TL;DR

UMAL (Uncountable Mixture of Asymmetric Laplacians) is a deep learning framework designed to predict the full, complex probability distribution of a target variable rather than just a single point estimate. By treating the output as an infinite mixture of "Asymmetric Laplacian" components, it captures multi-modal and skewed data—common in real-world scenarios like finance and real estate—without the rigid assumptions of traditional models.

What Problem Does This Paper Address?

In standard regression, we usually ask a model to predict a single value (e.g., "What is the price of this house?"). However, real-world data is often "messy" or heterogeneous. A single house might have a price distribution that is skewed, has multiple peaks (modes), or varies wildly depending on hidden factors.

Existing methods to handle this "aleatoric uncertainty" (the inherent randomness in data) have significant drawbacks:

  1. Simple Parametric Models: Assuming a Normal (Gaussian) distribution fails when the data is skewed or multi-modal.
  2. Mixture Density Networks (MDNs): These combine several distributions but require the user to guess the number of components beforehand, which can lead to unstable training.
  3. Quantile Regression (QR): While flexible, it can suffer from "crossing quantiles," where the predicted 70th percentile is lower than the 50th percentile—a physical impossibility.

Methods

The authors introduce UMAL, which builds on the Asymmetric Laplacian Distribution (ALD). The ALD is a mathematical tool closely linked to quantile regression; minimizing quantile loss is equivalent to maximizing the likelihood of an ALD.

The "secret sauce" of UMAL is the Uncountable Mixture. Instead of picking a fixed number of components (like 3 or 10), the model treats the output as an integral over an infinite number of ALDs, indexed by an asymmetry parameter $\tau$ (ranging from 0 to 1).

UMAL Architecture and Output Generation

How it works in Deep Learning: * The neural network takes the original features ($x$) plus a randomly sampled value of $\tau$ as inputs. * The network predicts the parameters ($\mu, b$) for that specific $\tau$. * During training, the model uses Monte Carlo integration: it samples several $\tau$ values, calculates the loss for each, and optimizes the weights to maximize the overall log-likelihood of the mixture. * This allows a finite-parameter neural network to implicitly represent an infinite mixture of functions.

Datasets

The researchers evaluated UMAL on four distinct types of data:

UMAL Predictions on Synthetic Data
  1. Synthetic Data: A toy dataset with four different regimes (Beta distribution, Normal, Uniform, and Multimodal) to test if the model could adapt to changing distribution shapes.
  2. Room Price Forecasting (RPF): Public Airbnb listing data from Barcelona and Vancouver. This represents a high-variance real-world task.
  3. Financial Estimation: A large, private dataset of bank account transactions used to predict monthly spending and income.
  4. UCI Benchmarks: Standard regression datasets used to verify calibration quality.

Key Insights and Ideas

  • Solving the "Crossing" Problem: Because UMAL treats the different quantiles as part of a single mixture distribution rather than independent estimates, it naturally avoids the "crossing quantiles" problem that plagues standard quantile regression.
  • Implicit Complexity: By feeding $\tau$ into the network as an input, the model learns a continuous mapping. This means it doesn't just learn a few "types" of distributions; it learns the entire spectrum of the output's shape.
  • Shape as a Feature: One of the most interesting ideas is using the shape of the predicted distribution for further analysis. The authors used t-SNE and DBSCAN to cluster houses in Barcelona based on their price distribution shapes, revealing geographical patterns in how prices fluctuate.

Key Results

UMAL consistently outperformed standard baselines (Normal, Laplace, and MDNs) across all datasets:

  • Log-Likelihood: In the Barcelona Airbnb dataset, UMAL achieved a log-likelihood of -18.04, significantly better than the Laplace distribution (-19.84) or a 10-component Normal MDN (-27.72).
  • Calibration: The model is "well-calibrated," meaning its probability estimates are trustworthy. If UMAL says there is a 90% chance a price falls within a certain range, it actually does so in the test data with high precision (error as low as 0.04).
  • Stability: Unlike MDNs, which often crashed or gave poor results when the number of components was set incorrectly, UMAL proved robust across different tasks.

Limitations and Future Work

  • Aleatoric vs. Epistemic: The authors note that UMAL focuses strictly on aleatoric uncertainty (noise in the data). It does not account for epistemic uncertainty (uncertainty due to a lack of data or model ignorance). They suggest that combining UMAL with Bayesian techniques (like Dropout or Bayes-by-Backprop) would be a powerful next step.
  • Inference Cost: To get the "full" distribution during prediction, you have to pass multiple $\tau$ values through the network. While efficient, it is computationally heavier than a simple point-estimate regression.
  • Architecture Agnosticism: While the framework is flexible, the paper primarily uses standard dense and LSTM layers. Exploring how UMAL interacts with more complex architectures (like Transformers) remains an open area.

Summary updated

Abstract

In regression tasks, aleatoric uncertainty is commonly addressed by considering a parametric distribution of the output variable, which is based on strong assumptions such as symmetry, unimodality or by supposing a restricted shape. These assumptions are too limited in scenarios where complex shapes, strong skews or multiple modes are present. In this paper, we propose a generic deep learning framework that learns an Uncountable Mixture of Asymmetric Laplacians (UMAL), which will allow us to estimate heterogeneous distributions of the output variable and shows its connections to quantile regression. Despite having a fixed number of parameters, the model can be interpreted as an infinite mixture of components, which yields a flexible approximation for heterogeneous distributions. Apart from synthetic cases, we apply this model to room price forecasting and to predict financial operations in personal bank accounts. We demonstrate that UMAL produces proper distributions, which allows us to extract richer insights and to sharpen decision-making.
BibTeX
@inproceedings{brando2019modelling,
  author = {Axel Brando and José A. Rodríguez-Serrano and Jordi Vitrià and Alberto Rubio},
  title = {{Modelling heterogeneous distributions with an Uncountable Mixture of Asymmetric Laplacians}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=J1eadPTMPC},
}
Figure