On Minimum Distance Estimation for Binomial Regression Models

Boxiao LiuUniversity 2016 · DOI

TL;DR

This thesis introduces two novel robust estimation methods, Minimum Hellinger Distance Estimation (MHDE) and Symmetric Chi-squared Distance Estimation (SCDE), for binomial regression models. These methods offer improved performance over traditional Maximum Likelihood Estimation (MLE) when data is contaminated or the model is misspecified, making them valuable for real-world applications where perfect data is rare.

What Problem Does This Paper Address?

In statistical modeling, especially for binary outcomes (like yes/no, success/failure), Maximum Likelihood Estimation (MLE) is a popular go-to. However, MLE is notoriously sensitive to "bad" data – outliers or slight deviations from the assumed model. This sensitivity can lead to unreliable estimates, which is a significant problem in fields like medicine, toxicology, and social sciences where data imperfections are common. The paper addresses this by developing and evaluating new estimation techniques that are more resilient to these issues, aiming for both efficiency (good performance when data is clean) and robustness (stable performance when data is imperfect).

Methods

The core of this work lies in proposing and analyzing two minimum distance estimation methods:

  • Minimum Hellinger Distance Estimation (MHDE): This method aims to minimize the Hellinger distance between the observed data distribution and the model's predicted distribution. The Hellinger distance is a metric that measures the similarity between two probability distributions, and minimizing it often leads to robust estimators.
  • Symmetric Chi-squared Distance Estimation (SCDE): Similar to MHDE, SCDE minimizes a different distance measure, the symmetric chi-squared distance, between the observed and predicted distributions. This approach also aims to provide robustness.

The paper delves into the theoretical properties of these estimators, including their consistency and asymptotic normality, to understand their behavior. Crucially, it then uses extensive Monte Carlo simulations to compare MHDE and SCDE against the standard MLE under various scenarios, including different link functions (logistic and normal CDFs) and simulated data contamination.

Datasets

Not discussed in the paper. The paper uses simulated data generated according to specific models and parameters to evaluate the performance of the proposed estimators.

Key Insights and Ideas

  • Robustness is Crucial: The paper strongly emphasizes that real-world data is rarely perfect, and robust estimation methods are essential for reliable statistical inference.
  • MHDE's Equivalence to MLE (in ideal conditions): Theoretically, MHDE is shown to be asymptotically equivalent to MLE when the model is correctly specified, meaning it performs just as well as MLE when data is clean.
  • MHDE and SCDE Outperform MLE in Contaminated Scenarios: Simulation results consistently show that both MHDE and SCDE are more robust than MLE when data is contaminated (e.g., with outliers or skewed distributions). They often exhibit smaller biases and mean squared errors in these situations.
  • Link Function Matters: The choice of link function (like logistic or normal CDF) influences the performance of the estimators, highlighting the importance of model selection.
  • Contamination Type Affects Performance: The type of contamination (e.g., long-tailed vs. short-tailed) impacts how well the robust estimators perform relative to MLE, with MHDE and SCDE showing particular strengths against long-tailed contamination.

Key Results

The simulation studies reveal several key findings:

  • MHDE and SCDE are generally more robust: Across various contamination models and parameter settings, MHDE and SCDE consistently showed better performance (lower bias and/or MSE) compared to MLE, especially when the data was contaminated.
  • MHDE is asymptotically efficient: When the model is correctly chosen, MHDE is shown to be asymptotically equivalent to MLE, meaning it's efficient.
  • Performance varies with contamination: MHDE and SCDE demonstrated particular robustness against long-tailed contamination and strict-increase scenarios. In some cases, they significantly outperformed MLE.
  • Specific performance differences: For instance, in some scenarios with logistic CDF and specific parameter values, MHDE showed smaller biases than MLE, and in others, SCDE had smaller MSEs. The paper provides detailed tables (e.g., Table 2.1, 3.1, 4.1) showcasing these comparisons.

Limitations and Future Work

  • Simulated Data: The study relies on simulated data. While this allows for controlled experiments, future work could focus on applying these methods to real-world datasets to validate their performance in practice.
  • Specific Distributions: The simulations primarily used logistic and normal CDFs as link functions. Future research could explore the performance of MHDE and SCDE with a wider range of continuous and discrete distributions.
  • Parameter Space Exploration: The paper suggests that the location and scale parameters of the CDF can affect estimator performance. Further investigation into how these parameters interact with the estimators is recommended.
  • Complex Covariate Structures: The current study assumes a relatively straightforward covariate structure. Investigating the performance with more complex or high-dimensional covariate scenarios would be valuable.

Summary updated

Abstract

This thesis investigates efficient and robust estimators for binomial regression models. For this purpose, I have made use of two minimum distance estimation methods developed for discrete data, namely Minimum Hellinger Distance Estimation (MHDE) and Symmetric Chi-squared Distance Estimation (SCDE) methods. These methods generally known to produce efficient estimators when the chosen model is correct and, at the same time, are robust against model misspecification and outliers. Asymptotic properties and robustness features of the proposed estimators are discussed through theoretical demonstrations and simulations. Furthermore, the performance of estimators is compared with the traditional estimation approach of the maximum likelihood estimation. Binomial regression models generally requires a specified “link function.” In this thesis, cumulative distribution functions of the logistic and standard normal distributions are primarily used as the link functions. From theoretical results, it is concluded that the proposed MHDE is asymptotically equivalent to the maximum likelihood estimator when the model is correctly chosen. Some asymptotic properties of the proposed SCDE estimator is studied. Monte Carlo simulations are carried out compare the estimators for small to moderate sample sizes. It is observed that both MHDE and SCDE estimators show some robustness against model contamination, and the MHDE and the SCDE outperform the MLE in various conditions. Optimal conditions are discussed through extensive simulations under different scenarios.
BibTeX
@inproceedings{liu2016minimum,
  author = {Boxiao Liu},
  title = {{On Minimum Distance Estimation for Binomial Regression Models}},
  booktitle = {University},
  year = {2016},
  doi = {10.7939/r3v40k692},
}
Figure