Making Classical Machine Learning Pipelines Differentiable: A Neural Translation Approach

Gyeong-In Yu, Saeed Amizadeh, Artidoro Pagnoni, Byung-Gon Chun, Markus Weimer, Matteo InterlandiarXiv 2019 · OpenReview

Introduction

Classical machine learning (ML) pipelines, such as those utilizing Gradient Boosted Trees (GBDTs) or Support Vector Machines (SVMs), are staples in industrial applications due to their robustness and performance on tabular data. However, these pipelines are typically constructed as Directed Acyclic Graphs (DAGs) of independent operators trained in a "greedy," sequential manner. This lack of end-to-end differentiability prevents the early stages of a pipeline from learning from the errors of later stages, potentially leaving significant performance gains on the table.

This paper presents a framework to translate entire classical ML pipelines into functionally equivalent, differentiable neural networks. By doing so, the entire system can be jointly optimized via backpropagation, combining the structural strengths of classical models with the global optimization capabilities of deep learning.

The Neural Translation Framework

The core of the approach is "Neural Translation," where pre-trained classical operators are converted into neural network layers.

Translating Decision Trees

The most challenging aspect of this translation is handling non-differentiable logic, such as the hard splits in decision trees. The authors approximate these using smooth functions:

  • Internal Nodes: Split conditions (e.g., $x_3 > 5.1$) are replaced with neurons using sigmoid activation functions.
  • Leaf Activations: Each leaf is represented as a conjunction of these smooth split functions.
  • Output: The final prediction is a weighted sum of leaf activations.
Figure 1: Translation of a decision tree into a differentiable neural network architecture.

Categorical Features and Embeddings

Standard preprocessing steps like one-hot or hash encoding are translated into trainable embedding layers. This allows the model to learn dense, low-dimensional representations of categorical data that are optimized specifically for the downstream task during fine-tuning.

Figure 6: Translation of categorical feature encodings into trainable embedding matrices.

Fine-Tuning Strategies

The framework supports varying levels of "trainability." For instance, one might only update leaf values (Level 1) to preserve the tree's original logic while allowing for minor adjustments, or treat the entire translated structure as a fully-connected Multi-Layer Perceptron (Level 4) for maximum flexibility.

Key Insights

  • Warm Start Advantage: Initializing the neural network with weights from a pre-trained classical model ("Warm Start") significantly outperforms random initialization. The classical model provides a "principled" architecture and a strong starting point, leading to faster convergence and better local minima.
  • Natural Regularization: For smaller datasets, restricting the number of trainable parameters (e.g., only tuning leaf values) acts as an effective regularizer, preventing the overfitting often seen in standard deep learning models on tabular data.
  • Architecture Search: Rather than guessing a neural network architecture, the structure of a successful GBDT serves as a data-driven blueprint for the network's design.

Experimental Results

The framework was evaluated on large-scale tabular datasets including Credit (310K records), Flight (21M records), and Criteo (46M records).

The results demonstrate that translated and fine-tuned models consistently outperform both the original classical pipelines and standard MLP baselines. On the Criteo dataset, a LightGBM pipeline's AUC improved from 0.7756 to 0.8102 after translation and joint optimization of embeddings and tree structures. This approach also outperformed specialized architectures like DeepGBM, while offering greater flexibility.

Conclusion

By bridging the gap between classical ML and deep learning, this neural translation approach allows practitioners to leverage the best of both worlds. It transforms static, greedily-trained pipelines into dynamic, end-to-end differentiable systems capable of reaching superior performance levels through joint optimization.

Reference: Yu, G. I., et al. (2019). Making Classical Machine Learning Pipelines Differentiable: A Neural Translation Approach. arXiv:1906.03822v2.

Summary updated

Abstract

Classical Machine Learning (ML) pipelines often comprise of multiple ML models where models, within a pipeline, are trained in isolation. Conversely, when training neural network models, layers composing the neural models are simultaneously trained using backpropagation. We argue that the isolated training scheme of ML pipelines is sub-optimal, since it cannot jointly optimize multiple components. To this end, we propose a framework that translates a pre-trained ML pipeline into a neural network and fine-tunes the ML models within the pipeline jointly using backpropagation. Our experiments show that fine-tuning of the translated pipelines is a promising technique able to increase the final accuracy.
BibTeX
@misc{yu2019making,
  author = {Gyeong-In Yu and Saeed Amizadeh and Artidoro Pagnoni and Byung-Gon Chun and Markus Weimer and Matteo Interlandi},
  title = {{Making Classical Machine Learning Pipelines Differentiable: A Neural Translation Approach}},
  year = {2019},
  eprint = {1906.03822},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.03822},
}
Figure