Abstract
We reveal the phenomenon that “naive” multivariate local polynomial regression can adapt to local smooth lower dimensional structure in the sense that it achieves the optimal convergence rate for nonparametric estimation of regression functions belonging to a Sobolev space when the predictor variables live on or close to a lower dimensional manifold.
Introduction
In nonparametric regression, the "curse of dimensionality" poses a significant challenge: as the number of predictor variables ($D$) increases, the data density required for accurate estimation grows exponentially. Consequently, the convergence rate of standard estimators typically degrades in high-dimensional settings. However, real-world high-dimensional data often exhibits a hidden, lower-dimensional structure—for instance, points in a 100D space might actually lie on a 2D curved manifold.
Traditional approaches address this by first estimating the manifold (dimensionality reduction) and then performing regression on the reduced representation. This paper investigates whether a "naive" multivariate local polynomial regression, applied directly to the high-dimensional data without explicit manifold estimation, can still achieve the optimal convergence rate dictated by the data's intrinsic dimension.
Methodology
The authors focus on Local Polynomial Regression, specifically local linear regression. To estimate a function at point $x$, a weighted linear regression is performed using data within a neighborhood defined by a bandwidth $h$.
Crucially, the "naive" approach uses the full observed dimension ($D$) in the regression formula. To ensure optimality, the method incorporates:
- Intrinsic Dimension Estimation: Utilizing a maximum likelihood estimator (developed by Levina and Bickel) to determine the true dimension $d$.
- Modified Generalized Cross-Validation (mGCV): A blockwise cross-validation technique to select the optimal bandwidth $h$, which must be tuned according to the intrinsic dimension $d$ rather than the observed dimension $D$.
Experimental Setup
The theoretical findings were validated using synthetic numerical experiments. The primary setup involved a 3D predictor vector $X = (X_1, X_2, X_3)$, where $X_2$ and $X_3$ are deterministic functions of $X_1$, creating a 1D manifold (a curve) embedded in 3D space.
To test robustness, varying levels of Gaussian noise were added to the coordinates, displacing points slightly from the manifold. The "blind" multivariate regression (using all 3 variables) was compared against an "oracle" univariate regression (using only $X_1$).
Results and Insights
The study yielded several key insights:
- Theoretical Optimality: The authors proved that the Mean Squared Error (MSE) achieves a convergence rate of $n^{-2p/(2p+d)}$, the optimal rate for a function with smoothness $p$ on a $d$-dimensional space.
- Automatic Adaptation: The variance of the estimator scales with $h^{-d}$ rather than $h^{-D}$, indicating that the algorithm naturally ignores irrelevant dimensions.
- Geodesic vs. Euclidean Distance: For smooth manifolds, Euclidean distance is locally proportional to geodesic distance (distance along the surface). This allows the "naive" algorithm to function effectively without explicit surface mapping.
- Noise Tolerance: The "blind" estimator remained stable even as noise increased. As long as the noise scale is small relative to the bandwidth ($o(h)$), the optimal convergence rate is maintained.
In simulations, the intrinsic dimension estimator $\hat{d}$ was highly accurate (e.g., $\hat{d} \approx 1.023$ for $d=1$), and the multivariate estimator performed nearly as well as the oracle univariate model.
Discussion
The primary advantage of this "blind" approach is avoiding the computational overhead and potential errors of explicit manifold learning algorithms like ISOMAP or LLE. However, the method still relies on an accurate estimate of the intrinsic dimension $d$ for bandwidth selection. Future work could explore performance on non-smooth manifolds (e.g., those with singularities) and the global stability of bandwidth selection across regions with varying intrinsic dimensions.
This summary is based on the research by Bickel and Li.
Summary updated