Abstract
This paper addresses the challenge of statistical inference for regression coefficients in high-dimensional Single-Index Models (SIMs). By leveraging the properties of elliptically symmetric designs, we demonstrate that a SIM can be reformulated as a proxy linear model where the regression parameter remains identifiable. We develop an estimation procedure analogous to the debiased Lasso in standard linear models, achieving root-n-consistency and asymptotic normality without requiring the estimation of the unknown link function. For Gaussian designs, we further enhance efficiency by expanding the link function through a Hermite polynomial basis. The validity and performance of our approach are demonstrated through extensive simulation experiments in high-dimensional settings.
Introduction
In modern statistical analysis, researchers frequently encounter datasets where the number of features ($p$) significantly exceeds the number of observations ($n$). Simultaneously, the relationship between these features and the response variable is often non-linear. The Single-Index Model (SIM) serves as a powerful tool in these scenarios, assuming the response $y$ depends on a linear combination of features $x$ (the "index") through an unknown non-linear link function $g$:
$$E[y|x] = g(\langle x, au \rangle)$$
Traditionally, performing inference—such as constructing confidence intervals or calculating p-values for the coefficients $\tau$—requires a non-parametric estimation of $g$. In high-dimensional spaces, this estimation is notoriously difficult, often requiring restrictive smoothness assumptions or large sample sizes that are unavailable. This paper proposes a method to perform valid inference on $\tau$ while remaining entirely agnostic about the specific form of the link function $g$.
Methodology
The Proxy Linear Model
The core innovation of this work lies in the use of a "proxy linear model." The authors prove that if the covariates $x$ follow an elliptically symmetric distribution (such as a Gaussian distribution), the SIM can be treated as a linear model:
$$Y = X\beta + z$$
In this formulation, $\beta$ is a scaled version of the true parameter $\tau$. Crucially, under the assumption of elliptical symmetry, the predictors $X$ and the "noise" term $z$ (which encapsulates the non-linear components of the original model) are uncorrelated. This orthogonality allows for the application of linear estimation techniques to a non-linear problem.
Debiased Lasso for SIMs
To handle the high-dimensional nature of the data ($p > n$), the authors adapt the debiased Lasso. The process involves:
- Initial Estimation: Obtaining a sparse pilot estimate of the coefficients using the Lasso.
- Bias Correction: Applying a projection of the residuals to remove the bias inherent in $L_1$ regularization.
- Sample Splitting: Utilizing a sample-splitting regime to ensure that the estimation of the covariance structure does not interfere with the final inference step, maintaining the integrity of the resulting p-values.
Efficiency via Hermite Polynomials
While the proxy linear model provides a valid path for inference, treating the non-linearity as noise can be inefficient. For Gaussian designs, the authors propose an enhanced estimator. By expanding the unknown link function $g$ using the Hermite polynomial basis, the model can capture more of the non-linear signal. This reduces the variance of the estimator and leads to narrower confidence intervals without sacrificing the "link-agnostic" nature of the primary approach.
Simulation Results
The methodology was validated through simulations designed to mimic high-dimensional challenges, typically setting $p = 2000$ and $n = 1000$. The experiments tested various non-linear link functions, including the sign function and exponential functions, across different covariance structures (e.g., independent features and AR(1) correlation).
Key findings include:
- Asymptotic Normality: The estimators consistently exhibited root-n-consistency, with error distributions following the predicted bell curve.
- Coverage Probability: The debiased estimators achieved empirical coverage rates for 95% confidence intervals that were very close to the nominal level.
- Variance Reduction: In Gaussian settings, employing a 5th-order Hermite expansion significantly improved efficiency, reducing the Mean Squared Error (MSE) by over 50% in certain non-linear scenarios compared to the simple linear proxy.
Discussion and Limitations
The primary advantage of this framework is its "link-agnosticism." Researchers can obtain valid inference for regression coefficients without knowing if the underlying model is a logistic regression, a probit model, or a more complex non-linear relationship.
However, the method relies heavily on the assumption of elliptical symmetry in the feature distribution. While this is a common assumption in theoretical frameworks and controlled experiments (like compressed sensing), it may be violated in observational data where features are highly skewed or have complex dependencies. Furthermore, the model assumes the link function is independent of the covariates. Future research may focus on extending these debiasing techniques to non-symmetric designs and exploring alternative basis functions to accommodate a broader range of data distributions.
Summary updated