Machine learning and multinomal models: A simulation study

Félix Chan, Mark N. Harris, Rohit Singh, W Yeo2023 · DOI

Abstract

In recent times, the use of machine learning in academic research has become an increasingly popular option in the field of econometric research. A significant amount of literature has pitched machine learning techniques for improving variable selection or causal effect evaluation. However, conversely, there is an equal amount of literature warning against blindly making use of these techniques without adjustments. In order to attain a better understanding of how these estimators behave compared to traditional counterparts, this paper conducts a Monte Carlo simulation study between shrinkage-based techniques (LASSO and Elastic Net) and traditional Logistical Regression in a sparse multinomial model environment. The study compares selection consistency and estimator variance across different sample sizes. Overall, the results show that in sparse multinomial environments, the Elastic Net provides a clear advantage in selection consistency, particularly in small-sample scenarios. However, this advantage is not universal, as it introduces a natural bias in coefficient assignment.

Introduction: The Econometric Crossroads

Econometrics is currently at a crossroads regarding the integration of machine learning (ML). While some researchers champion ML for its ability to handle complex variable selection and causal inference, others warn against the "blind" application of these tools without a deep understanding of their statistical properties. This paper specifically targets the sparse multinomial environment—a common scenario where researchers attempt to predict categorical outcomes (e.g., consumer choice among multiple brands) using many potential variables, only a few of which are truly relevant.

Methodology: The Monte Carlo Approach

To evaluate these methods, we employed a Monte Carlo simulation, generating synthetic datasets where the "ground truth" parameters are known. This allows for a direct comparison of how well different models recover the true underlying structure. We compared three primary estimators:

  1. Logit (Maximum Likelihood): The traditional statistical baseline.
  2. LASSO (Least Absolute Shrinkage and Selection Operator): An ML technique that penalizes coefficient size to shrink irrelevant variables to zero.
  3. Elastic Net: A hybrid approach combining LASSO and Ridge regression penalties.

The study varied sample sizes from 500 to 4,000 and tested variables with different signal strengths: "strong" (coefficient = 0.2), "weak" (coefficient = 0.005), and "irrelevant" (coefficient = 0).

Selection Consistency: The Small Data Advantage

The most striking finding is the performance of the Elastic Net in "small data" scenarios. Selection consistency—the probability of correctly identifying a relevant variable—is significantly higher for the Elastic Net when the sample size is limited (n=500).

Figure 1: Selection consistency for a strong variable (X1, Class 1) across different sample sizes.

As shown in Figure 1, the Elastic Net (blue dashed line) starts with a much higher consistency at lower sample sizes compared to traditional Logit and Maximum Likelihood estimators. While traditional methods eventually catch up as the sample size grows to 4,000, the Elastic Net reaches near-perfect consistency much faster. Furthermore, shrinkage methods proved vastly superior at noise suppression, consistently zeroing out irrelevant variables that traditional Logit often mistakenly identified as significant.

The Bias-Variance Trade-off

While the Elastic Net excels at variable selection, it faces a classic statistical hurdle: the bias-variance trade-off. Because shrinkage methods work by penalizing the magnitude of coefficients, they inherently underestimate the true impact of a variable.

Figure 6: Distribution of coefficient estimates showing the bias-variance trade-off between Logit and Elastic Net.

Figure 6 illustrates this trade-off clearly. The traditional Logit estimator (tan) is centered on the true value of 0.2, indicating it is unbiased, but it exhibits a wide spread (high variance). In contrast, the Elastic Net (teal) and the Max Lik variant (purple) show a much tighter distribution (low variance) but are centered around a biased value of approximately 0.12. This systematic underestimation makes these methods potentially problematic for researchers who require precise point estimates for policy decisions.

Conclusion

This study suggests that for econometricians primarily interested in variable selection—identifying what matters rather than the exact magnitude of the effect—the Elastic Net is a powerful upgrade over traditional tools, especially when data is scarce. However, the struggle to identify weakly relevant variables remains an open challenge for all tested methods. Future research should expand this comparison to more complex models like Random Forests or Neural Networks within the same multinomial context to see if they can overcome the inherent biases of basic shrinkage techniques.

Summary updated

Abstract

In recent times, the use of machine learning in academic research has become a more and more attractive option, an option that is becoming increasingly popular in the field of econometric research A significant amount of econometric literature has pitched the use of machine learning techniques for the purposes of improving research in roles such as variable selection or causal effect evaluation, in papers such as Athey and Imbens (2017) and Varian (2014).However, conversely, there is an equal amount of literature warning against blindly making use of these techniques without adjustments in papers such as Athey and Imbens ( 2019) and Mullainathan and Spiess (2017).In order to attain a better understanding of how the estimators provided by some of these machine learning techniques behave compared to their more traditional counterparts, this paper conducts a Monte Carlo Simulation study between Shrinkage based techniques such as LASSO and the Elastic Net, against Logistical Regression in a sparse multinomial model environment, comparing both selection consistency and estimator variance between different sample sizes.Overall, the results of this paper show that when dealing with sparse multinomial environments, of the methods tested, the Elastic Net provides the clearest advantage in selection consistency, both being able to mark strongly relevant variables as strongly relevant more often than LASSO and Logit, but also managing to avoid marking irrelevant variables as relevant.Moreover, when compared to its peers Elastic Net holds the greatest advantage when dealing with datasets with low samples.However, this advantage is not universal, as it holds a natural bias when assigning values to coefficients and its ability to pick out weakly relevant variables is comparable to that of Logit.
BibTeX
@misc{chan2023machine,
  author = {Félix Chan and Mark N. Harris and Rohit Singh and W Yeo},
  title = {{Machine learning and multinomal models: A simulation study}},
  year = {2023},
  doi = {10.36334/modsim.2023.chan70},
}
Figure