Likelihood Adaptively Modified Penalties

Yang Feng, Tengfei Li, Zhiliang Ying

Abstract

A new family of penalty functions, adaptive to likelihood, is introduced for model selection in general regression models. It arises naturally through assuming certain types of prior distribution on the regression parameters. To study stability properties of the penalized maximum likelihood estimator, two types of asymptotic stability are defined. Theoretical properties, including the parameter estimation consistency, model selection consistency, and asymptotic stability, are established under suitable regularity conditions. An efficient coordinate-descent algorithm is proposed. Simulation results and real data analysis show that the proposed method has competitive performance in comparison with existing ones.

Introduction

In high-dimensional statistics, researchers frequently encounter datasets where the number of variables ($p$) far exceeds the number of observations ($n$). To identify the most relevant factors, penalty functions are employed to shrink unimportant coefficients to zero. Historically, this field has been divided between convex penalties (like LASSO), which are computationally stable but biased, and non-convex penalties (like SCAD and MCP), which reduce bias but can be numerically unstable due to local minima.

Likelihood Adaptively Modified Penalties (LAMP) bridge this gap by proposing that the penalty should not be a generic function, but rather one tailored to the specific likelihood of the regression model (e.g., Logistic, Poisson, or Gamma).

The LAMP Framework

The LAMP family is derived from a Bayesian perspective using a Negative Absolute Prior (NAP). The core innovation is Likelihood-Specific Design: if a generalized linear model (GLM) is defined by a link function $g(\xi)$, the LAMP penalty is constructed using that same function $g$. This ensures the penalty's mathematical properties—such as concavity and smoothness—complement the log-likelihood function.

Comparison of penalty functions (left) and their derivatives (right) for LASSO, SCAD, MCP, and LAMP variants (Sigmoid and Poisson).

As shown in the figure above, LAMP penalties (Sigmoid and Poisson) offer a distinct profile compared to traditional methods. Unlike SCAD and MCP, which have discontinuous derivatives, LAMP penalties are often infinitely differentiable. This smoothness creates a more stable optimization landscape, allowing for more consistent numerical solutions.

Theoretical Stability and Smoothness

A key contribution of this work is the definition of Weak and Strong Asymptotic Stability. These frameworks rigorously measure how much an estimator fluctuates when the underlying data is slightly perturbed. By aligning the "shape" of the penalty with the "shape" of the loss function, LAMP achieves a superior balance between sparsity and stability.

Illustration of the alignment between the log-likelihood and LAMP (Sigmoid) versus MCP.

By carefully selecting tuning parameters, the authors demonstrate that the penalized likelihood remains locally convex around the true parameters. This allows LAMP to enjoy the low bias characteristic of non-convex penalties without the typical instability associated with them.

Empirical Performance

The authors evaluated LAMP using both synthetic simulations and high-dimensional biological datasets:

  • Simulated Data: In logistic and Poisson regression simulations, the "Sigmoid" version of LAMP consistently achieved lower False Positive rates than SCAD and MCP while maintaining high True Positive detection rates.
  • Lung Cancer Data: On a gene expression dataset (12,533 genes), LAMP identified a highly accurate model using only 5 genes, whereas other methods required 13–15 genes for similar performance.
  • Prostate Cancer Data: LAMP produced the most parsimonious model, selecting only 23 genes compared to LASSO's 30.

Conclusion and Future Work

LAMP introduces a robust framework for model selection that prioritizes the relationship between the penalty and the model's likelihood. While the current implementation focuses on individual variable selection in GLMs, future research will explore extending LAMP to grouped variable selection and more complex semiparametric regression models. Additionally, refining the computational efficiency of tuning the concavity parameter ($\lambda_0$) remains a priority for scaling to even larger datasets.

Summary updated

Abstract

A new family of penalty functions, adaptive to likelihood, is introduced for model selection in general regression models. It arises naturally through assuming certain types of prior distribution on the regression parameters. To study stability properties of the penalized maximum likelihood estimator, two types of asymptotic stability are defined. Theoretical properties, including the parameter estimation consistency, model selection consistency, and asymptotic stability, are established under suitable regularity conditions. An efficient coordinate-descent algorithm is proposed. Simulation results and real data analysis show that the proposed method has competitive performance in comparison with existing ones.
BibTeX
@inproceedings{feng2018likelihood,
  author = {Yang Feng and Tengfei Li and Zhiliang Ying},
  title = {{Likelihood Adaptively Modified Penalties}},
  booktitle = {Applied Stochastic Models in Business and Industry},
  year = {2018},
  doi = {10.1002/asmb.2322},
}
Figure