A Deep Neural Network Two-part Model and Feature Importance Test for Semi-continuous Data

Baiming Zou, Xinlei Mi, James G. Xenakis, Di Wu, Jianhua Hu, Fei ZoubioRxiv 2023 · DOI

TL;DR

Researchers have developed a new deep learning framework, sDNN, specifically designed for "semi-continuous" data—datasets characterized by a large number of zeros and a skewed distribution of positive values (like pain scores or medical expenses). By combining a two-part neural network architecture with a novel "bagging and filtering" stability mechanism and a permutation-based feature importance test (fsDNN), the model significantly outperforms traditional statistical methods in both prediction accuracy and interpretability.

What Problem Does This Paper Address?

In clinical and economic studies, we often encounter "semi-continuous" data. Imagine a study on postoperative pain: many patients feel no pain at all (score = 0), while others experience varying degrees of intensity (score > 0). This creates a "spike at zero" that breaks standard regression models.

Traditionally, statisticians use "two-part models": a logistic regression to predict if an event happens (zero vs. non-zero) and a linear model to predict the intensity of that event. However, these traditional models rely on "linear additive" assumptions—they assume variables don't interact in complex ways. In reality, factors like age, surgery type, and anesthesia interact non-linearly. While standard Deep Neural Networks (DNNs) can model these complexities, they are notoriously unstable on the smaller, noisier datasets typical of clinical trials.

Methods

The authors propose two interconnected methods:

  1. sDNN (Stable DNN): This is a two-part architecture where one DNN handles the binary classification (zero vs. positive) and another handles the continuous prediction (how positive?). To solve the instability of DNNs, they use Bootstrap Aggregating (Bagging)—training many models on different subsets of data. Crucially, they add a Filtering Procedure: they rank the models based on out-of-bag performance and only keep the "top performers" for the final ensemble, effectively pruning out models that fell into poor local minima during training.
  2. fsDNN (Feature Importance sDNN): To peek inside the "black box," the authors developed a permutation-based testing procedure. By shuffling a specific feature and measuring how much the model's error increases, they calculate a feature importance score. They use a cross-fitting strategy (splitting data into training and validation sets) to ensure that the importance scores are statistically valid and not just artifacts of overfitting.

Datasets

The model was evaluated using:

  • Real-world Clinical Data: A study of 7,391 surgical patients assessing acute postoperative pain (POP). The dataset included 37 variables such as anesthesia method, surgery duration, age, gender, and BMI.
  • Simulated Scenarios: The authors created four complex simulation environments using different distributions (Truncated-normal, Log-normal, and Poisson) to test the model against varying levels of non-linearity and "nuisance" variables.

Key Insights and Ideas

  • "Many could be better than all": The core philosophy behind their filtering algorithm is that in an ensemble of neural networks, including poorly converged models actually hurts the final prediction. Pruning the bottom-tier models leads to a more robust "stable" DNN.
  • The Curse of Dimensionality: Traditional semi-parametric models (like single-index models) often fail when the number of variables increases because they struggle to estimate complex link functions. The sDNN bypasses this by leveraging the universal approximation power of deep layers.
  • Feature Selection as a Performance Booster: The authors found that using fsDNN to identify and then re-train the model on only the most important features (feature selection) actually improved predictive performance beyond the initial sDNN.

Key Results

  • Superior Accuracy: On the real-world pain data, sDNN achieved a classification AUC of 0.935, compared to 0.867 for traditional parametric models.
  • Massive Error Reduction: In complex simulations, sDNN reduced the Mean Squared Prediction Error (MSPE) by 65% to 81% compared to traditional methods.
  • Stability: The filtering mechanism successfully mitigated the "randomness" of neural network initialization, providing consistent results across different data splits.
  • Interpretability: The fsDNN successfully identified known clinical drivers of pain (like surgery type and duration) while providing a statistically rigorous p-value for each feature.

Limitations and Future Work

While powerful, the current framework has a few boundaries:

  • Correlated Data: The model assumes observations are independent. It does not yet account for longitudinal data (the same patient measured multiple times) or clustered data (patients within the same hospital).
  • Computational Cost: Training an ensemble of 100+ DNNs with cross-validation is significantly more computationally expensive than a simple logistic regression.
  • High-Dimensional Scaling: While it handles 37–100 variables well, the authors suggest that further refinement is needed for "ultra-high-dimensional" data (e.g., genomics with thousands of features).

The authors have released an R package for the community to implement these methods, available on GitHub.

Summary updated

Abstract

Semi-continuous data frequently arise in clinical practice. For example, while many surgical patients suffer from varying degrees of acute postoperative pain (POP) post surgery (i.e., POP score > 0), others experience none (i.e., POP score = 0), indicating the existence of two distinct data processes at play. Existing parametric or semi-parametric two-part modeling methods for this type of semicontinuous data can fail to appropriately model these two underlying data processes as such methods rely heavily on (generalized) linear additive assumptions. However, many factors may interact to jointly influence the experience of POP non-additively and non-linearly. Motivated by this challenge and inspired by the flexibility of deep neural networks (DNN) to accurately approximate complex functions universally, we derive a DNN-based two-part model by adapting the conventional DNN methods by adding two additional components: a bootstrapping procedure along with a filtering algorithm to boost the stability of the conventional DNN, an approach we denote as sDNN. To improve the interpretability and transparency of sDNN, we further derive a feature importance testing procedure to identify important features contributing to the outcome measurements of the two data processes, denoting this approach fsDNN. We show that fsDNN not only offers a valid feature importance test but also that using the identified features can further improve the predictive performance of sDNN. The proposed sDNN- and fsDNN-based twopart models are applied to the analysis of real data from a POP study, in which application they clearly demonstrate advantages over the existing parametric and semi-parametric two-part models. Further, we conduct extensive numerical studies to demonstrate that sDNN and fsDNN consistently outperform the existing two-part models regardless of the data complexity. An R package implementing the proposed methods has been developed and deposited on GitHub ( https://github.com/SkadiEye/fsDNN ).
BibTeX
@misc{zou2023deep,
  author = {Baiming Zou and Xinlei Mi and James G. Xenakis and Di Wu and Jianhua Hu and Fei Zou},
  title = {{A Deep Neural Network Two-part Model and Feature Importance Test for Semi-continuous Data}},
  year = {2023},
  doi = {10.1101/2023.06.07.544106},
  howpublished = {bioRxiv},
}
Figure