On the benefits of maximum likelihood estimation for Regression and Forecasting

Pranjal Awasthi, Abhimanyu Das, Rajat Sen, Ananda Theertha Suresh

Abstract

Traditional machine learning workflows often rely on Target Metric Optimization (TMO), where models are trained to directly minimize a specific evaluation metric such as Mean Squared Error (MSE) or Mean Absolute Error (MAE). This paper challenges this paradigm, demonstrating that Maximum Likelihood Estimation (MLE)—learning the underlying probability distribution of the data—is theoretically more robust and practically superior. By capturing the data's structure, a single MLE-trained model can be adapted post-hoc to optimize for multiple metrics at inference time, often outperforming specialized TMO models on complex, real-world datasets.

Introduction: The Limitations of Target Metric Optimization

In regression and forecasting, the standard approach is to align the training loss function with the final evaluation metric. If the goal is low MSE, practitioners typically use Least Squares; for low MAE, they minimize absolute deviation. While intuitive, TMO suffers from two primary drawbacks:

  1. Rigidity: A separate model must be trained for every desired metric.
  2. Suboptimality in "Messy" Data: For datasets involving counts (Poisson) or heavy-tailed distributions (Pareto), direct metric optimization can lead to poor generalization compared to models that explicitly model the data's distribution.

Methodology: A Two-Step MLE Approach

The authors propose a flexible alternative to TMO that decouples learning from metric optimization:

  1. Training: Use MLE to learn the parameters of a conditional probability distribution $p(y|x; \theta)$.
  2. Inference: Once the distribution is learned, calculate the specific statistic that minimizes the target metric. For example, output the mean for MSE or the median for MAE.

The ZNBP Mixture Likelihood

To handle diverse data types without manual distribution selection, the researchers introduced the ZNBP mixture likelihood family. This model combines Zero-inflation (for sparse data), Negative Binomial (for bursty count data), and Pareto (for heavy-tailed extreme values) components. The model automatically learns the appropriate weights for these components during training, adapting to the specific "shape" of the dataset.

Empirical Evaluation

The proposed MLE (ZNBP) approach was evaluated against several TMO baselines across diverse datasets, including the M5 (Walmart sales) and Favorita (grocery sales) forecasting challenges, as well as regression tasks like Bicycle Share and Gas Turbine emissions.

Comparison of MLE (ZNBP) against Target Metric Optimization (TMO) baselines on Favorita and M5 datasets.

As shown in Figure 1, the single MLE (ZNBP) model is highly competitive. On the Favorita dataset, it achieves the best performance in both MAPE and WAPE. Notably, the table highlights the suboptimality of TMO: on Favorita, the model specifically trained for MSE (TM0(MSE)) actually performs worse on the RMSE metric than the model trained for MAE or the MLE model. This supports the authors' claim that direct optimization can be less robust than distributional learning in the presence of complex data.

Key Insights and Theoretical Contributions

  • Competitiveness Guarantee: The paper provides a theoretical framework proving that MLE is "competitive" with any other estimator. If a good estimation for a data property exists, MLE is guaranteed to find a solution that is nearly as effective.
  • Robustness to Outliers: In heavy-tailed settings (e.g., Pareto regression), the authors prove that MLE provides better excess risk bounds than TMO, making it less susceptible to being skewed by outliers.
  • Efficiency: Because the model learns the full distribution, business requirements can change post-training (e.g., switching from minimizing average error to minimizing extreme misses) without requiring a full model retrain.

Conclusion

This research suggests that the choice of loss function is as critical as network architecture. By moving from point-prediction TMO to distributional MLE, practitioners can build models that are more flexible, theoretically grounded, and robust to the nuances of real-world data distributions. While complex mixture likelihoods introduce non-convexity and slight computational overhead during inference, the gains in generalization and versatility make MLE a compelling standard for modern regression and forecasting tasks.

Summary updated

BibTeX
@inproceedings{awasthi2022benefits,
  author = {Pranjal Awasthi and Abhimanyu Das and Rajat Sen and Ananda Theertha Suresh},
  title = {{On the benefits of maximum likelihood estimation for Regression and Forecasting}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=zrW-LVXj2k1},
}
Figure