Abstract
Semi-structured regression (SSR) models jointly learn the effect of structured (tabular) and unstructured (non-tabular) data through additive predictors and deep neural networks (DNNs), respectively. Inference in SSR models aims at deriving confidence intervals for the structured predictor, although current approaches ignore the variance of the DNN estimation of the unstructured effects. This results in an underestimation of the variance of the structured coefficients and, thus, an increase of Type-I error rates. To address this shortcoming, we present here a theoretical framework for structured inference in SSR models that incorporates the variance of the DNN estimate into confidence intervals for the structured predictor. By treating this estimate as a random offset with known variance, our formulation is agnostic to the specific deep uncertainty quantification method employed. Through numerical experiments and a practical application on a medical dataset, we show that our approach results in increased coverage of the true structured coefficients and thus a reduction in Type-I error rate compared to ignoring the variance of the neural network, naive ensembling of SSR models, and a variational inference baseline.
Introduction: The Challenge of Semi-Structured Regression
In fields like medicine, researchers often need to combine "structured" data (such as a patient's age or blood pressure) with "unstructured" data (such as an MRI scan or clinical notes). Semi-structured regression (SSR) models facilitate this by using a Deep Neural Network (DNN) for unstructured inputs and a linear or additive model for tabular features.
However, a critical problem arises during statistical inference. Standard models provide p-values and confidence intervals to determine if a feature (e.g., age) significantly affects an outcome. Current SSR approaches typically treat the DNN’s output as a fixed, perfect value. In reality, DNNs are uncertain. Ignoring this "shakiness" in the neural network component leads to an underestimation of the total variance, making tabular features appear more significant than they actually are, thereby increasing the Type-I error (false positive) rate.
Methodology: Accounting for Neural Network Variance
The proposed framework treats the DNN's estimate not as a fixed number, but as a random offset with known variance. This allows the model to account for the "Structured Effect" and "Unstructured Effect" jointly while acknowledging the uncertainty inherent in the deep learning component.
Key technical contributions include:
- Exact Inference for Linear Models: The authors derive an unbiased estimator for structured coefficients ($\beta$) and its exact variance, incorporating the DNN's covariance matrix ($\Gamma$).
- Extension to GLMs and GAMs: By utilizing the framework of Generalized Linear Mixed Models (GLMMs), the method performs inference for non-Gaussian responses (like binary or count data) by treating the DNN's uncertainty as a random effect.
- Plug-and-Play DUQ: The framework is agnostic to the specific Deep Uncertainty Quantification (DUQ) method used. Whether using Deep Ensembles, Monte Carlo Dropout, or Laplace approximations, the framework only requires a mean and a variance estimate.
- Two-Step Procedure: To avoid "artificial shrinkage" or bias, the authors recommend training the DNN first and then performing inference on a held-out validation set.
Experimental Validation
The paper validates the theory using three distinct setups:
- Synthetic Poisson Data: A controlled environment to test theoretical properties where the "ground truth" uncertainty is known.
- MNIST-based Simulation: Tabular features are combined with MNIST digit images to simulate a semi-structured task with real-world image complexity.
- ISIC 2020 Challenge Dataset: A real-world medical dataset containing 32,531 dermoscopic images of skin lesions alongside patient metadata (age, sex, anatomy site) to predict malignancy.
Key Results and Performance
The primary finding is that accounting for DNN uncertainty is vital for maintaining nominal coverage. As shown in the results, "uncertainty-unaware" models see their confidence interval coverage drop significantly as DNN uncertainty increases. In contrast, the proposed method consistently hits the target coverage levels.
Additional highlights include:
- Reduced False Positives: The method significantly reduced Type-I error rates compared to naive ensembling or variational inference (Bayes-by-Backprop).
- Real-World Impact: On the ISIC melanoma dataset, the proposed model achieved an adjusted $R^2$ of 0.31, significantly outperforming the uncertainty-unaware baseline ($R^2 = 0.11$). It also produced larger, more realistic p-values for features like "age," preventing overconfident clinical claims.
- Better Predictions: Interestingly, accounting for uncertainty also improved predictive accuracy. By treating the DNN output as a random effect, the model can "correct" for biased neural network predictions, reducing RMSE by over 20% in some simulations.
Discussion and Future Directions
A key insight from the study is the "dilution" effect: while the specific uncertainty of the DNN becomes less critical in the large-sample limit, it is vital for the small-to-medium datasets common in clinical research.
The primary limitation is that the framework is only as good as the underlying DUQ method. If the method used to estimate the DNN's variance is poorly calibrated, those errors will propagate into the structured inference. Future research will focus on benchmarking which DUQ methods pair best with this framework across different data modalities and disentangling epistemic uncertainty from aleatoric noise.
Summary updated