GrCAN: Gradient Boost Convolutional Autoencoder with Neural Decision Forest

Manqing Dong, Lina Yao, Xianzhi Wang, Boualem Benatallah, Shuai ZhangarXiv 2018 · arXiv

Abstract

GrCAN (Gradient Boost Convolutional Autoencoder with Neural Decision Forest) is a novel architecture that fuses the feature-extraction power of Convolutional Autoencoders with the robust classification of Neural Decision Forests. By adding a Gradient Boosting module to learn from prediction residuals, the model achieves state-of-the-art classification performance across both small-scale UCI datasets and high-dimensional image data like Fashion-MNIST.

Introduction

In the machine learning landscape, a persistent divide exists between Deep Neural Networks (DNNs) and Random Forests (RFs). DNNs excel at handling high-dimensional, raw data through hierarchical feature learning but are prone to overfitting on small datasets and require extensive tuning. Conversely, Random Forests are robust and effective on small datasets but struggle with high-dimensional raw inputs and lack differentiability, hindering end-to-end integration. GrCAN addresses this by creating a unified, end-to-end trainable model that combines the representation learning of CNNs, the robustness of decision trees, and the additive power of gradient boosting.

Methods

The GrCAN architecture integrates three primary components into a single pipeline:

  1. Input Processing (Convolutional Autoencoder): A Convolutional Autoencoder (CAE) is used to learn a compressed, "clean" hidden representation ($H$) of the input. The decoder ensures these features are meaningful by reconstructing the original input, acting as a regularizer during the feature learning phase.
  2. Neural Decision Forest (NDF): The CAE features are fed into a differentiable version of a Random Forest. Unlike traditional trees with hard splits, NDFs utilize stochastic routing via sigmoid functions at decision nodes, making the entire forest compatible with backpropagation.
  3. Gradient Boost (GB) Module: This core innovation embeds a GB module to specifically learn the residual error of the initial NDF prediction. The final output is the sum of the NDF's prediction and the GB module's correction.
The full GrCAN architecture, integrating a Convolutional Autoencoder for feature extraction with a Neural Decision Forest and Gradient Boost module.

The training process is sequential: the model first optimizes the CAE and fully connected layers, then the leaf node distributions of the forest, and finally the parameters of the gradient boost modules. This sequential optimization leads to more stable convergence compared to simultaneous training.

The sequential gradient boosting process within GrCAN, where successive modules learn from the residuals of previous predictions.

Results

GrCAN was evaluated on a diverse range of datasets, consistently outperforming standard CNNs, Random Forests, and GBDT.

  • Fashion-MNIST: GrCAN achieved an accuracy of 92.01%, surpassing standard CNNs (89.94%) and previous hybrid models like Peter et al. (85.64%).
  • Epileptic Seizure Recognition: The model reached 76.50% accuracy, a significant improvement over traditional GBDT (49.61%) and standard Random Forests (56.84%).
  • Small-scale Datasets (UCI): On datasets like Iris and Wine, the model achieved over 95% accuracy, demonstrating its ability to avoid overfitting on limited data.

While the addition of GB modules increases computational complexity per iteration, the authors noted that the model reaches target accuracy in fewer total iterations compared to non-boosted versions.

Conclusion

GrCAN successfully bridges the gap between symbolic logic and continuous optimization. By applying residual learning to differentiable forests, it provides a robust and powerful framework for both low- and high-dimensional classification tasks. Future work may explore extending this architecture to regression tasks and further optimizing the trade-off between boosting complexity and training time.

Summary updated

Abstract

Random forest and deep neural network are two schools of effective classification methods in machine learning. While the random forest is robust irrespective of the data domain, the deep neural network has advantages in handling high dimensional data. In view that a differentiable neural decision forest can be added to the neural network to fully exploit the benefits of both models, in our work, we further combine convolutional autoencoder with neural decision forest, where autoencoder has its advantages in finding the hidden representations of the input data. We develop a gradient boost module and embed it into the proposed convolutional autoencoder with neural decision forest to improve the performance. The idea of gradient boost is to learn and use the residual in the prediction. In addition, we design a structure to learn the parameters of the neural decision forest and gradient boost module at contiguous steps. The extensive experiments on several public datasets demonstrate that our proposed model achieves good efficiency and prediction performance compared with a series of baseline methods.
BibTeX
@misc{dong2018grcan,
  author = {Manqing Dong and Lina Yao and Xianzhi Wang and Boualem Benatallah and Shuai Zhang},
  title = {{GrCAN: Gradient Boost Convolutional Autoencoder with Neural Decision Forest}},
  year = {2018},
  eprint = {1806.08079},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.08079},
}
Figure