Self-born Wiring for Neural Trees

Ying Chen, Feng Mao, Jie Song, Xinchao Wang, Huiqiong Wang, Mingli Song

Introduction

Deep Neural Networks (DNNs) have become the standard for high-accuracy image classification due to their ability to learn complex features automatically. However, they are often criticized for being "black boxes"—computationally expensive and difficult to interpret. Conversely, Decision Trees (DTs) offer high interpretability and efficiency but struggle with high-dimensional data like raw images.

Self-born Wiring (SeBoW) is a novel framework designed to bridge this gap. It transforms standard DNNs into efficient, interpretable "neural trees." Unlike previous methods that grow trees greedily or use rigid structures, SeBoW utilizes a "construction-by-destruction" approach, allowing a "mother" network to autonomously rewire itself into a globally optimized tree structure.

Methodology: Construction-by-Destruction

The SeBoW framework follows a multi-stage process to derive a neural tree from a pre-existing high-performing architecture, such as VGG-13.

The SeBoW pipeline: from a mother network (VGG-13) to a neural tree via unpacking and installing routers.
  1. Destruction (Unpacking): The mother network is partitioned into sections, and each layer is decomposed into isolated "filter groups" or base learners.
  2. Construction (Wiring): Two types of routers are introduced to manage data flow:
  3. Senders: Soft-max classifiers that determine which learners in the subsequent layer receive data.
  4. Receivers: Utilizing the Gumbel-Softmax trick, these components make discrete routing decisions differentiable, enabling the use of standard gradient descent for wiring optimization.
  5. Optimization: The process involves a search phase to identify the optimal wiring, followed by a retraining phase where the discovered tree structure is fine-tuned to maximize accuracy.

Experimental Evaluation

SeBoW was evaluated across several benchmarks, ranging from small-scale datasets to the massive ImageNet challenge.

  • CIFAR-10 & CIFAR-100: Used for initial validation and comparison against baseline architectures.
  • ImageNet: Serving as the gold standard, SeBoW's success on this dataset (1.33 million images) marks a significant milestone for neural tree research, which has historically struggled with scalability.

Key Insights and Semantic Hierarchy

A core advantage of SeBoW is its ability to learn a logical, semantic hierarchy. Rather than arbitrary splits, the model naturally learns to separate broad categories (e.g., animals vs. vehicles) at higher levels of the tree, refining these into specific classes (e.g., cars vs. trucks) at deeper levels.

Visualization of the search space and the resulting searched structure, highlighting the semantic class separation.

This sparsity is not just a byproduct but a feature; the router design naturally prunes unnecessary connections, leading to a lightweight final model that offers both visual interpretability and computational efficiency.

Results and Performance

SeBoW consistently outperforms previous hybrid models and remains competitive with standard non-tree architectures:

  • Efficiency: On CIFAR-10, SeBoW-C achieved 94.24% accuracy with only 4.6M parameters, significantly fewer than the 11.2M parameters required by a standard ResNet-18.
  • Scalability: On ImageNet, SeBoW-C reached a 70.13% Top-1 accuracy, surpassing its mother network (VGG-13) while maintaining a more efficient structure.
  • Inference Speed: The model supports a "single-path" inference mode, which executes only a fraction of the total neurons, providing substantial speedups over traditional dense networks.

Conclusion

SeBoW represents a significant advancement in the development of scalable and interpretable neural trees. By leveraging global differentiable optimization, it avoids the local optima common in greedy growth methods. While currently dependent on a pre-defined mother architecture, future research aims to expand this framework to more diverse tasks like object detection and segmentation.

Summary updated

Abstract

Neural trees aim at integrating deep neural networks and decision trees so as to bring the best of the two worlds, including representation learning from the former and faster inference from the latter. In this paper, we introduce a novel approach, termed as Self-born Wiring (SeBoW), to learn neural trees from a mother deep neural network. In contrast to prior neural-tree approaches that either adopt a pre-defined structure or grow hierarchical layers in a progressive manner, task-adaptive neural trees in SeBoW evolve from a deep neural network through a construction-by-destruction process, enabling a global-level parameter optimization that further yields favorable results. Specifically, given a designated network configuration like VGG, SeBoW disconnects all the layers and derives isolated filter groups, based on which a global-level wiring process is conducted to attach a subset of filter groups, eventually bearing a lightweight neural tree. Extensive experiments demonstrate that, with a lower computational cost, SeBoW outperforms all prior neural trees by a significant margin and even achieves results on par with predominant non-tree networks like ResNets. Moreover, SeBoW proves its scalability to large-scale datasets like ImageNet, which has been barely explored by prior tree networks.
BibTeX
@inproceedings{chen2021self,
  author = {Ying Chen and Feng Mao and Jie Song and Xinchao Wang and Huiqiong Wang and Mingli Song},
  title = {{Self-born Wiring for Neural Trees}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00500},
}
Figure