Introduction
Missing data is a ubiquitous challenge in machine learning, particularly in high-stakes domains like healthcare and sensor-based IoT. Standard approaches typically rely on imputation—filling in missing values with means, medians, or K-Nearest Neighbors—or training models designed to ignore missing values. However, these methods often fail to capture the underlying relationships between features. If a model treats every input feature as an independent entity, losing one feature results in the total loss of the information it carries.
This research introduces a simple yet powerful architectural modification to make neural networks more resilient to missing data. By grouping statistically correlated features into clusters and processing them through "Split Neural Networks," the model learns to utilize redundant information from available features to compensate for missing ones.
Methodology
The proposed approach consists of a two-stage pipeline designed to encode the correlation structure of the data directly into the model's topology.
1. Hierarchical Feature Clustering
Instead of clustering data points, the authors cluster the features themselves. They use Pearson correlation distance to measure similarity between features. Using hierarchical clustering with complete linkage, the $d$ input features are grouped into $k$ exhaustive clusters.
2. Split Neural Networks (Split NNs)
Rather than feeding all features into a single dense layer, the architecture is "split." Each feature cluster is fed into its own dedicated sub-network, referred to as a "deep split."
- Proportional Scaling: The number of hidden units in each split is proportional to the number of features in that specific cluster.
- Joint Training: All splits are eventually merged and trained using a joint loss function (categorical cross-entropy for classification or Mean Squared Error for regression).
- Imputation: While simple mean imputation is used for missing values, the Split NN architecture performs the heavy lifting by extracting robust representations from the remaining features within the cluster.
Experimental Evaluation
The researchers evaluated the Split NN approach on eight benchmark datasets, primarily from the UCI Machine Learning Repository. These datasets represent real-world scenarios in healthcare and biology where missingness is common, with missing data percentages ranging from 0.25% to 43.7%.
- Classification Tasks: bands, kidney disease, hepatitis, horse, mammographics, pima, and winconsin.
- Regression Task: Life Expectancy (WHO) dataset.
Results and Insights
The core innovation of this work is treating redundancy as a safety net. By forcing the network to process correlated features together in a dedicated sub-structure, the model learns a representation of the cluster rather than just individual features. This makes the model naturally robust; if one feature in a cluster is missing, other features in the same split provide the necessary context to maintain performance.
Key Findings:
- State-of-the-Art Performance: On several datasets, including "bands," "hepatitis," and "pima," the Split NN approach outperformed established methods like MICE (Multiple Imputation by Chained Equations) and specialized architectures.
- Significant Gains in Regression: In the Life Expectancy task—which featured a high missingness rate of 43.7%—the Split NN achieved a Test RMSE of 4.006, significantly outperforming the Vanilla NN's 5.116.
- Architectural Inductive Bias: By splitting the network, the authors effectively encode the "correlation structure" of the data into the model's topology, allowing for more graceful handling of unseen missingness at test time.
Conclusion and Future Work
Split Neural Networks offer a simple, effective way to improve model robustness without the complexity of generative models like GANs. While the method shows clear advantages, future work could explore the sensitivity of the cluster count ($k$) and whether combining Split NNs with more sophisticated imputation techniques (like KNN) would yield further gains. Additionally, the scalability of managing many network splits for high-dimensional data remains an open area for investigation.
Summary updated