On the benefits of maximum likelihood estimation for Regression and Forecasting

Pranjal Awasthi, Abhimanyu Das, Rajat Sen, Ananda Theertha Suresh

TL;DR

This paper argues that Maximum Likelihood Estimation (MLE) is often a better choice for training regression and forecasting models than directly optimizing for a specific target metric (like MSE or MAE). By using a flexible "mixture likelihood" family, the authors show that a single MLE-trained model can outperform specialized models across multiple metrics and handle complex data distributions like heavy tails and zero-inflation.

What Problem Does This Paper Address?

In supervised learning, the standard practice is Target Metric Optimization (TMO): if you want your model to perform well on Mean Squared Error (MSE), you train it using a squared loss function. If you want to minimize Mean Absolute Error (MAE), you use an L1 loss.

However, this approach has two major drawbacks. First, it is rigid; you must train a separate model for every metric you care about. Second, TMO can be fragile. When data is "messy"—containing heavy tails (outliers) or many zeros (sparsity)—standard loss functions like MSE can produce suboptimal results. The authors investigate whether we can do better by focusing on modeling the underlying data distribution itself rather than just the error metric.

Methods

The authors advocate for a practical Maximum Likelihood Estimation (MLE) approach. Instead of predicting a single point value, the model predicts the parameters of a probability distribution.

1. The "ZNBP" Mixture Likelihood

To make MLE practical for diverse real-world data, the authors propose a general-purpose mixture likelihood family called ZNBP, which combines three components:

  • Zero-inflation: A constant at zero to handle sparse, "bursty" data.
  • Negative Binomial: To capture the behavior of count data.
  • Pareto: To robustly handle heavy-tailed distributions and outliers.

2. Post-hoc Inference

Because the model learns the full distribution, it becomes "metric agnostic." At inference time, you can extract different statistics from the same model to optimize different metrics. For example, to minimize MSE, you use the mean of the predicted distribution; to minimize MAE, you use the median.

3. Theoretical Competitiveness

The paper provides a "competitiveness" theorem, proving that under general conditions, the MLE approach is theoretically guaranteed to perform as well as any other estimator for a given property (up to logarithmic factors). They specifically show that for Poisson and Pareto regression, MLE provides better theoretical error bounds than standard least-squares regression.

Datasets

The authors evaluated their approach on four diverse datasets:

  • Forecasting: The M5 (Walmart sales) and Favorita (grocery sales) datasets, both of which involve complex hierarchies and intermittent demand.
  • Regression: The Bicycle Share dataset (predicting rental counts) and the Gas Turbine dataset (predicting sensor measurements).

Key Insights and Ideas

  • One Model, Many Metrics: The most striking insight is that a single MLE model can often beat models specifically trained for a single metric. For instance, the MLE model often achieved a lower MAPE (Mean Absolute Percentage Error) than a model trained directly on the MAPE loss.
  • Baking in Inductive Bias: MLE allows practitioners to incorporate domain knowledge. If you know your data consists of counts or has extreme outliers, you can choose a likelihood (like Negative Binomial or Pareto) that reflects that reality, giving the optimizer a "head start."
  • Robustness to Heavy Tails: The authors demonstrate that in heavy-tailed settings (Pareto regression), MLE is naturally more robust than TMO. While TMO (least squares) struggles with the high variance of outliers, MLE's focus on the distribution parameters keeps it stable.

Key Results

The empirical results show that the MLE approach with the ZNBP mixture is remarkably consistent.

Performance comparison of MLE vs. TMO across different metrics on forecasting datasets.

As seen in Figure 1, the MLE(ZNBP) model is almost always the best or second-best performer across all metrics (MAPE, WAPE, RMSE) for both the Favorita and M5 datasets. Notably, on the Favorita dataset, the MLE model achieved a MAPE of 0.3139, outperforming the TMO(MAPE) model which scored 0.3199. This suggests that modeling the distribution is a more effective way to reach the metric's optimum than optimizing the metric directly.

In the regression tasks, the results were even more pronounced. On the Bicycle Share dataset, the MLE model outperformed all TMO baselines by a significant margin across every single metric.

Limitations and Future Work

The authors acknowledge a few hurdles:

  • Non-convexity: Introducing complex mixture likelihoods can make the loss function non-convex. While the authors found that standard deep learning optimizers (like Adam) handled this well in practice, it could theoretically lead to optimization difficulties in some settings.
  • Likelihood Selection: While the ZNBP mixture is versatile, MLE still requires some choice of distribution. If the chosen family is completely mismatched with the data, performance will suffer.

Future research directions include applying this general competitiveness framework to other domains beyond regression, such as classification with structured outputs or reinforcement learning.

Summary updated

Abstract

We advocate for a practical Maximum Likelihood Estimation (MLE) approach towards designing loss functions for regression and forecasting, as an alternative to the typical approach of direct empirical risk minimization on a specific target metric. The MLE approach is better suited to capture inductive biases such as prior domain knowledge in datasets, and can output post-hoc estimators at inference time that can optimize different types of target metrics. We present theoretical results to demonstrate that our approach is competitive with any estimator for the target metric under some general conditions. In two example practical settings, Poisson and Pareto regression, we show that our competitive results can be used to prove that the MLE approach has better excess risk bounds than directly minimizing the target metric. We also demonstrate empirically that our method instantiated with a well-designed general purpose mixture likelihood family can obtain superior performance for a variety of tasks across time-series forecasting and regression datasets with different data distributions.
BibTeX
@inproceedings{awasthi2022benefits,
  author = {Pranjal Awasthi and Abhimanyu Das and Rajat Sen and Ananda Theertha Suresh},
  title = {{On the benefits of maximum likelihood estimation for Regression and Forecasting}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=zrW-LVXj2k1},
}
Figure