Introduction
Missing data is a pervasive challenge in machine learning, often arising from sensor failures, incomplete surveys, or data entry errors. Traditional approaches typically rely on imputation—filling in missing values using statistical estimates like the mean or k-nearest neighbors. However, these methods can fail to capture complex data distributions or introduce noise that degrades performance during inference. Researchers from MIT and BITS Pilani have proposed a structural solution: the Split Neural Network (Split NN) architecture, which leverages feature correlations to build inherent robustness to missing information.
The Split Neural Network Architecture
The core innovation lies in a two-step pipeline designed to localize the impact of missing data through architectural design rather than complex pre-processing.
- Hierarchical Clustering: Instead of treating input features as a single flat vector, the model uses hierarchical clustering based on Pearson correlation distance. This groups statistically related features (e.g., systolic and diastolic blood pressure) into distinct clusters.
- Split NNs: The network is partitioned into sub-networks, or "deep splits," each dedicated to a specific feature cluster. The number of hidden units in each split is scaled according to the cluster size.
- Joint Training: These sub-networks are trained simultaneously using a joint loss function (categorical cross-entropy for classification or mean squared error for regression).
By isolating correlated features into separate pathways, the model creates internal redundancy. If a feature is missing, its sub-network can still derive meaningful signals from its correlated counterparts within the same cluster, effectively providing a proxy for the missing data. This structural robustness allows the model to remain effective even when using simple mean imputation.
Experimental Evaluation
The researchers evaluated the Split NN on eight benchmark datasets with varying degrees of missingness, ranging from 0.25% (Wisconsin breast cancer) to 43.7% (WHO Life Expectancy). The model was compared against standard "Vanilla" neural networks and several state-of-the-art imputation-based methods.
Performance in Classification
In classification tasks, the Split NN consistently outperformed or matched existing methods. For instance, on the hepatitis dataset, it achieved an accuracy of 0.849, a significant improvement over the Vanilla NN's 0.716 and other established baselines.
Performance in Regression
The robustness of the architecture was particularly evident in the high-missingness Life Expectancy regression task. The Split NN achieved a Test Root Mean Squared Error (RMSE) of 4.006 (with $k=2$ clusters), substantially lower than the 5.116 recorded for the standard Vanilla NN. This represents a substantial reduction in prediction error despite nearly half the data being missing.
Conclusion and Future Directions
The Split NN architecture demonstrates that structural design can be as effective as complex pre-processing for handling missing data. By utilizing redundancy through hierarchical clustering, the model remains resilient even with simple mean imputation.
Future research may explore the sensitivity of the model to the number of clusters ($k$) and investigate whether combining Split NNs with advanced generative imputation techniques (like GANs) could further enhance performance. Additionally, testing the scalability of the hierarchical clustering step on high-dimensional datasets remains an open area for investigation.
Summary updated