Abstract
In regression tasks, aleatoric uncertainty is commonly addressed by considering a parametric distribution of the output variable, which is based on strong assumptions such as symmetry or unimodality. These assumptions are too limited in scenarios where complex shapes, strong skews, or multiple modes are present. We propose a generic deep learning framework that learns an Uncountable Mixture of Asymmetric Laplacians (UMAL), which allows for the estimation of heterogeneous distributions of the output variable. Despite having a fixed number of parameters, the model can be interpreted as an infinite mixture of components, yielding a flexible approximation for complex distributions.
Introduction
Standard regression models typically aim for a single point estimate, often assuming that errors follow a simple parametric distribution like a Gaussian or Laplacian curve. However, real-world data is rarely so well-behaved. For instance, predicting the price of an Airbnb listing might reveal a distribution with multiple peaks based on luxury versus budget tiers, or significant skewness. This inherent randomness is known as aleatoric uncertainty.
Existing solutions like Mixture Density Networks (MDNs) require pre-specifying the number of components, which is difficult to tune. Quantile Regression (QR) offers more flexibility but often suffers from "crossing quantiles" and lacks a natural, valid probability density function. UMAL addresses these issues by treating the output as an infinite mixture of distributions.
Methodology: The UMAL Framework
The core of UMAL is the Asymmetric Laplacian Distribution (ALD). In this distribution, the location parameter $\mu$ corresponds to a specific quantile $\tau$. By minimizing a quantile loss, one essentially performs Maximum Likelihood Estimation for an ALD.
UMAL extends this by considering an uncountable mixture of ALDs across all possible quantiles $\tau \in (0, 1)$. The neural network architecture, $\phi(x, \tau)$, takes both input features $x$ and a sampled quantile $\tau$ as inputs, outputting parameters $(\mu_\tau, b_\tau)$ for that specific component. During training, the model samples random values of $\tau$ and minimizes the negative log-likelihood of the resulting mixture.
Experimental Evaluation
The model was evaluated on three distinct data types:
- Synthetic Regression: A dataset featuring four regimes: asymmetric, symmetric, uniform, and multimodal distributions.
- Room Price Forecasting (RPF): Predicting Airbnb prices in Barcelona and Vancouver based on listing features.
- Financial Estimation: Forecasting monthly spending and income in personal bank accounts.
Key Results and Insights
UMAL effectively "upgrades" Quantile Regression by treating quantiles as components of a mixture, bypassing the "crossing quantiles" problem and yielding a proper, differentiable density function.
- Superior Performance: UMAL achieved the highest log-likelihood in almost all tests. On synthetic data, it scored -28.14 compared to the best MDN at -39.03.
- Calibration: The model is highly calibrated; in the Barcelona Airbnb dataset, UMAL's calibration error was 0.04, significantly better than the Normal distribution's 0.12.
- Qualitative Shape: As shown in the synthetic results, the model successfully captures distinct "humps" in multimodal data and adapts its shape to the data's complexity automatically.
Conclusion
UMAL provides a flexible and robust framework for modelling complex aleatoric uncertainty without the need to pre-specify the number of modes. While it focuses on data randomness, future work could combine UMAL with Bayesian methods to account for epistemic uncertainty. Its ability to produce proper distributions allows for richer insights and sharper decision-making in real-world applications like financial forecasting and price prediction.
Summary updated