Regularization Learning Networks: Deep Learning for Tabular Datasets

Ira Shavitt, Eran SegalNeurIPS 2018 · OpenReview

TL;DR

While Deep Neural Networks (DNNs) dominate image and speech tasks, they often underperform compared to Gradient Boosting Trees (GBTs) on tabular datasets. This paper introduces Regularization Learning Networks (RLNs), which close this gap by automatically learning a unique regularization penalty for every single weight in the network. The result is an ultra-sparse, highly interpretable, and competitive deep learning model tailored for structured data.

The Challenge of Tabular Data

In the machine learning landscape, a clear divide exists: DNNs rule "unstructured" data (images, audio, text), while GBTs (like XGBoost) dominate "structured" or tabular data (spreadsheets, medical records).

The authors argue this discrepancy stems from the nature of the data representation. Images rely on distributed representations—changing a single pixel rarely alters the meaning of the image. Tabular data, however, often features non-distributed representations where a single feature (like a patient's age or a specific lab value) can drastically impact the outcome, while other features may just be noise.

Standard DNNs apply a uniform regularization penalty (like $L_1$ or $L_2$) to all weights. This blunt approach forces the network to treat all inputs with similar caution, which is suboptimal for tabular data. Ideally, we would tune a different regularization coefficient for every weight. However, doing so via standard cross-validation would require tuning millions of hyperparameters—a computationally intractable task.

Introducing Regularization Learning Networks (RLNs)

The core innovation of RLNs is a method to efficiently tune these millions of regularization hyperparameters using gradient descent during the training process itself.

  1. Counterfactual Loss ($L_{CF}$): The authors introduce a new loss function that asks a "what if" question: How much would the training loss have decreased if we had updated the weights using the current regularization coefficients?
  2. Gradient-Based Tuning: Because $L_{CF}$ is differentiable, the model can calculate the gradient of the loss with respect to the regularization coefficients ($\lambda$). This allows the network to "learn" which weights should be heavily penalized (noise) and which should be allowed to grow (signal).
  3. Simplex Projection: To prevent the regularization coefficients from simply dropping to zero (which would minimize the penalty but lead to overfitting), the authors project the coefficients onto a simplex. This creates a "zero-sum game": for the network to relax the penalty on an important edge, it must increase the penalty on others.
  4. Simultaneous Updates: The resulting RLN architecture updates its weights and its regularization penalties simultaneously in each training iteration.

Experimental Setup

The authors evaluated their method on a complex, real-world tabular task: predicting human traits from gut microbiome data. * Dataset: 2,574 healthy participants. Features included relative abundances of bacterial species and genes, alongside basic covariates like Age, Gender, and BMI. * Targets: Nine different health traits, including HbA1c (blood sugar), Cholesterol (HDL and LDL), and BMI.

Key Insights

  • Extreme Sparsity: RLNs don't just prune the network; they aggressively obliterate unnecessary connections. The models eliminated up to 99.8% of network edges and 82% of input features. This suggests that for tabular data, success relies on identifying the tiny fraction of connections that actually matter.
  • Bridging the "Tabular Gap": The paper demonstrates that standard DNNs fail on tabular data because they cannot handle high variability in feature importance. By allowing the network to "turn off" specific inputs at a granular level, RLNs mimic the feature-selection strengths of decision trees.
  • Interpretability: Because RLNs are so sparse, they are naturally more interpretable. Instead of a "black box" where every input contributes a tiny fraction, RLNs provide a clear map of which specific features drive a prediction.

Key Results

RLNs significantly outperformed standard DNNs, improving the explained variance ($R^2$) by a factor of 2.57x on average, and achieved results comparable to Gradient Boosting Trees, the gold standard for this type of data.

Performance comparison of GBT, RLN, LM, and DNN across various traits.

Furthermore, the researchers discovered a powerful synergy when combining these models. The best overall performance across almost all traits was achieved by an ensemble of RLNs and GBTs. This suggests that RLNs learn different, complementary patterns compared to tree-based models.

Performance of mixed ensembles, highlighting the superiority of the GBT+RLN combination.

Limitations and Future Work

  • Hyperparameter Sensitivity: While RLNs automate the tuning of millions of $\lambda$ values, they introduce a new high-level hyperparameter ($\theta$, the normalization factor) that still requires manual tuning via cross-validation.
  • Computational Overhead: Calculating the Counterfactual Loss adds computational complexity to each training step, though the authors note it remains efficient enough for large datasets.
  • Hybrid Data Types: The authors suggest RLNs could be the key to training "hybrid" models—for example, a single network that natively processes both a patient's medical images (unstructured) and their electronic health records (tabular). This remains a highly promising area for future experimental validation.

Summary updated

Abstract

Despite their impressive performance, Deep Neural Networks (DNNs) typically underperform Gradient Boosting Trees (GBTs) on many tabular-dataset learning tasks. We propose that applying a different regularization coefficient to each weight might boost the performance of DNNs by allowing them to make more use of the more relevant inputs. However, this will lead to an intractable number of hyperparameters. Here, we introduce Regularization Learning Networks (RLNs), which overcome this challenge by introducing an efficient hyperparameter tuning scheme which minimizes a new Counterfactual Loss. Our results show that RLNs significantly improve DNNs on tabular datasets, and achieve comparable results to GBTs, with the best performance achieved with an ensemble that combines GBTs and RLNs. RLNs produce extremely sparse networks, eliminating up to 99.8% of the network edges and 82% of the input features, thus providing more interpretable models and reveal the importance that the network assigns to different inputs. RLNs could efficiently learn a single network in datasets that comprise both tabular and unstructured data, such as in the setting of medical imaging accompanied by electronic health records. An open source implementation of RLN can be found at https://github.com/irashavitt/regularization<em>learning</em>networks.
BibTeX
@inproceedings{shavitt2018regularization,
  author = {Ira Shavitt and Eran Segal},
  title = {{Regularization Learning Networks: Deep Learning for Tabular Datasets}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=S1ZLadWd-S},
}
Figure