Introduction
Classical machine learning (ML) pipelines, such as those utilizing Gradient Boosted Trees (GBDTs) or Support Vector Machines (SVMs), are staples in industrial applications due to their robustness and performance on tabular data. However, these pipelines are typically constructed as Directed Acyclic Graphs (DAGs) of independent operators trained in a "greedy," sequential manner. This lack of end-to-end differentiability prevents the early stages of a pipeline from learning from the errors of later stages, potentially leaving significant performance gains on the table.
This paper presents a framework to translate entire classical ML pipelines into functionally equivalent, differentiable neural networks. By doing so, the entire system can be jointly optimized via backpropagation, combining the structural strengths of classical models with the global optimization capabilities of deep learning.
The Neural Translation Framework
The core of the approach is "Neural Translation," where pre-trained classical operators are converted into neural network layers.
Translating Decision Trees
The most challenging aspect of this translation is handling non-differentiable logic, such as the hard splits in decision trees. The authors approximate these using smooth functions:
- Internal Nodes: Split conditions (e.g., $x_3 > 5.1$) are replaced with neurons using sigmoid activation functions.
- Leaf Activations: Each leaf is represented as a conjunction of these smooth split functions.
- Output: The final prediction is a weighted sum of leaf activations.
Categorical Features and Embeddings
Standard preprocessing steps like one-hot or hash encoding are translated into trainable embedding layers. This allows the model to learn dense, low-dimensional representations of categorical data that are optimized specifically for the downstream task during fine-tuning.
Fine-Tuning Strategies
The framework supports varying levels of "trainability." For instance, one might only update leaf values (Level 1) to preserve the tree's original logic while allowing for minor adjustments, or treat the entire translated structure as a fully-connected Multi-Layer Perceptron (Level 4) for maximum flexibility.
Key Insights
- Warm Start Advantage: Initializing the neural network with weights from a pre-trained classical model ("Warm Start") significantly outperforms random initialization. The classical model provides a "principled" architecture and a strong starting point, leading to faster convergence and better local minima.
- Natural Regularization: For smaller datasets, restricting the number of trainable parameters (e.g., only tuning leaf values) acts as an effective regularizer, preventing the overfitting often seen in standard deep learning models on tabular data.
- Architecture Search: Rather than guessing a neural network architecture, the structure of a successful GBDT serves as a data-driven blueprint for the network's design.
Experimental Results
The framework was evaluated on large-scale tabular datasets including Credit (310K records), Flight (21M records), and Criteo (46M records).
The results demonstrate that translated and fine-tuned models consistently outperform both the original classical pipelines and standard MLP baselines. On the Criteo dataset, a LightGBM pipeline's AUC improved from 0.7756 to 0.8102 after translation and joint optimization of embeddings and tree structures. This approach also outperformed specialized architectures like DeepGBM, while offering greater flexibility.
Conclusion
By bridging the gap between classical ML and deep learning, this neural translation approach allows practitioners to leverage the best of both worlds. It transforms static, greedily-trained pipelines into dynamic, end-to-end differentiable systems capable of reaching superior performance levels through joint optimization.
Reference: Yu, G. I., et al. (2019). Making Classical Machine Learning Pipelines Differentiable: A Neural Translation Approach. arXiv:1906.03822v2.
Summary updated