TL;DR
This paper argues that Maximum Likelihood Estimation (MLE) is often a better choice for training regression and forecasting models than directly optimizing for a specific target metric (like MSE or MAE). By using a flexible "mixture likelihood" family, the authors show that a single MLE-trained model can outperform specialized models across multiple metrics and handle complex data distributions like heavy tails and zero-inflation.
What Problem Does This Paper Address?
In supervised learning, the standard practice is Target Metric Optimization (TMO): if you want your model to perform well on Mean Squared Error (MSE), you train it using a squared loss function. If you want to minimize Mean Absolute Error (MAE), you use an L1 loss.
However, this approach has two major drawbacks. First, it is rigid; you must train a separate model for every metric you care about. Second, TMO can be fragile. When data is "messy"—containing heavy tails (outliers) or many zeros (sparsity)—standard loss functions like MSE can produce suboptimal results. The authors investigate whether we can do better by focusing on modeling the underlying data distribution itself rather than just the error metric.
Methods
The authors advocate for a practical Maximum Likelihood Estimation (MLE) approach. Instead of predicting a single point value, the model predicts the parameters of a probability distribution.
1. The "ZNBP" Mixture Likelihood
To make MLE practical for diverse real-world data, the authors propose a general-purpose mixture likelihood family called ZNBP, which combines three components:
- Zero-inflation: A constant at zero to handle sparse, "bursty" data.
- Negative Binomial: To capture the behavior of count data.
- Pareto: To robustly handle heavy-tailed distributions and outliers.
2. Post-hoc Inference
Because the model learns the full distribution, it becomes "metric agnostic." At inference time, you can extract different statistics from the same model to optimize different metrics. For example, to minimize MSE, you use the mean of the predicted distribution; to minimize MAE, you use the median.
3. Theoretical Competitiveness
The paper provides a "competitiveness" theorem, proving that under general conditions, the MLE approach is theoretically guaranteed to perform as well as any other estimator for a given property (up to logarithmic factors). They specifically show that for Poisson and Pareto regression, MLE provides better theoretical error bounds than standard least-squares regression.
Datasets
The authors evaluated their approach on four diverse datasets:
- Forecasting: The M5 (Walmart sales) and Favorita (grocery sales) datasets, both of which involve complex hierarchies and intermittent demand.
- Regression: The Bicycle Share dataset (predicting rental counts) and the Gas Turbine dataset (predicting sensor measurements).
Key Insights and Ideas
- One Model, Many Metrics: The most striking insight is that a single MLE model can often beat models specifically trained for a single metric. For instance, the MLE model often achieved a lower MAPE (Mean Absolute Percentage Error) than a model trained directly on the MAPE loss.
- Baking in Inductive Bias: MLE allows practitioners to incorporate domain knowledge. If you know your data consists of counts or has extreme outliers, you can choose a likelihood (like Negative Binomial or Pareto) that reflects that reality, giving the optimizer a "head start."
- Robustness to Heavy Tails: The authors demonstrate that in heavy-tailed settings (Pareto regression), MLE is naturally more robust than TMO. While TMO (least squares) struggles with the high variance of outliers, MLE's focus on the distribution parameters keeps it stable.
Key Results
The empirical results show that the MLE approach with the ZNBP mixture is remarkably consistent.
As seen in Figure 1, the MLE(ZNBP) model is almost always the best or second-best performer across all metrics (MAPE, WAPE, RMSE) for both the Favorita and M5 datasets. Notably, on the Favorita dataset, the MLE model achieved a MAPE of 0.3139, outperforming the TMO(MAPE) model which scored 0.3199. This suggests that modeling the distribution is a more effective way to reach the metric's optimum than optimizing the metric directly.
In the regression tasks, the results were even more pronounced. On the Bicycle Share dataset, the MLE model outperformed all TMO baselines by a significant margin across every single metric.
Limitations and Future Work
The authors acknowledge a few hurdles:
- Non-convexity: Introducing complex mixture likelihoods can make the loss function non-convex. While the authors found that standard deep learning optimizers (like Adam) handled this well in practice, it could theoretically lead to optimization difficulties in some settings.
- Likelihood Selection: While the ZNBP mixture is versatile, MLE still requires some choice of distribution. If the chosen family is completely mismatched with the data, performance will suffer.
Future research directions include applying this general competitiveness framework to other domains beyond regression, such as classification with structured outputs or reinforcement learning.
Summary updated