Building Sparse Deep Feedforward Networks using Tree Receptive Fields

Xiaopeng Li, Zhourong Chen, Nevin L. Zhang

TL;DR

This paper introduces TRF-nets, a novel method for automatically learning sparse feedforward neural network (FNN) architectures. By leveraging tree-structured probabilistic models to identify correlated features and applying a "tree receptive field" concept inspired by CNNs, TRF-nets achieve comparable or better performance than dense FNNs with significantly fewer parameters and improved interpretability.

What Problem Does This Paper Address?

Designing effective neural network architectures is a notoriously difficult and time-consuming process, often referred to as "graduate student descent." While Convolutional Neural Networks (CNNs) excel at spatial data and Recurrent Neural Networks (RNNs) at sequential data, standard Feedforward Neural Networks (FNNs) are typically fully-connected, leading to a large number of parameters and potential overfitting. This paper tackles the challenge of automatically learning sparse, efficient, and interpretable FNN architectures from scratch, moving beyond traditional methods that rely on manual design or post-hoc pruning.

Methods

The core of the TRF-net approach is a "macro expansion" method that builds the network layer by layer.

  1. Learning Local Structure with Chow-Liu Trees: For the observed features at a given level, the authors first use the Chow-Liu algorithm to learn a tree-structured probabilistic graphical model. This tree captures pairwise dependencies and correlations between features.
  2. Building Tree Receptive Fields (TRFs): Based on the learned tree structure, "tree receptive fields" are defined. These fields identify subsets of strongly correlated features (neighbors in the tree). New "TRF neurons" are introduced, each connected to all units within a specific receptive field. This is analogous to how CNNs use local receptive fields, but adapted for the tree structure of feature dependencies.
  3. Introducing Global Neurons: To capture longer-range dependencies that might be missed by local TRFs, a small number of "global neurons" are also introduced, connected to all units in the previous layer.
  4. Layered Construction and Training: This two-layer structure (TRF neurons + global neurons) is trained as a denoising autoencoder. The hidden representations from this layer are then treated as new observed variables, and the process is repeated to build deeper TRF-nets. The entire network is ultimately trained using backpropagation.

Datasets

The paper evaluates TRF-nets on six public datasets:

  • Tox21 challenge dataset: A dataset for predicting toxic effects of chemical compounds, treated as 12 binary classification tasks.
  • Text classification datasets: Five datasets from Zhang et al. (2015): AG's News, DBPedia, Yelp Review, Yahoo! Answer, and Sogou News. These are represented using bag-of-words after preprocessing.

Key Insights and Ideas

  • Leveraging Probabilistic Models for Structure: The paper demonstrates that learning probabilistic dependencies (via Chow-Liu trees) can effectively guide the discovery of sparse, meaningful connections in FNNs, even for non-spatial/sequential data.
  • Tree Receptive Fields as a Novel Connectivity Pattern: The concept of "tree receptive fields" is a key innovation, adapting the successful local connectivity idea from CNNs to the structure learned from feature correlations. This allows for efficient representation of local interactions.
  • Interpretability as a Natural Outcome: The learned sparse structures and the localized nature of TRF neurons lead to more interpretable models, where hidden units can be characterized by groups of highly correlated input features (e.g., words). This is a significant advantage over dense FNNs.
  • Macro Expansion for Efficiency: Building the network layer by layer, evaluating only two-layer models at each step, is computationally more efficient than traditional "micro expansion" methods that add neurons one by one.

Key Results

  • Performance: TRF-nets achieved better or comparable classification accuracy/AUC compared to dense FNNs, pruned FNNs, and L1-regularized FNNs across all tested datasets.
  • Sparsity and Parameter Efficiency: TRF-nets achieved significantly fewer parameters and sparser structures (around 10% of connections compared to fully-connected FNNs) without sacrificing performance.
  • Interpretability: TRF-nets showed higher interpretability scores than FNNs and Pruned FNNs, with hidden units often correlating with semantically meaningful groups of words.
  • Model Depth: The TRF-net architecture proved robust to increasing depth, with performance sometimes improving with more layers, unlike deep FNNs which can be prone to overfitting.
  • Computational Time: While the structure learning phase of TRF-nets takes time, the overall computational cost, including hyperparameter tuning, was comparable to or better than methods requiring extensive manual grid search for FNN structures.

Limitations and Future Work

  • Dataset Specificity: The paper notes that the Sogou News dataset's vocabulary (Chinese) presented challenges for interpretability analysis using word2vec.
  • Hyperparameter Tuning: While easier than full structure search, TRF-nets still have hyperparameters like receptive field radius and stride that need to be tuned.
  • Future Work: The authors suggest exploring how model depth influences TRF-net performance further and potentially comparing with other macro expansion methods (though their implementation was unavailable). The interpretability aspect is highlighted as an important area for future research.

Summary updated

Abstract

Sparse connectivity is an important factor behind the success of convolutional neural networks and recurrent neural networks. In this paper, we consider the problem of learning sparse connectivity for feedforward neural networks (FNNs). The key idea is that a unit should be connected to a small number of units at the next level below that are strongly correlated. We use Chow-Liu's algorithm to learn a tree-structured probabilistic model for the units at the current level, use the tree to identify subsets of units that are strongly correlated, and introduce a new unit with receptive field over the subsets. The procedure is repeated on the new units to build multiple layers of hidden units. The resulting model is called a TRF-net. Empirical results show that, when compared to dense FNNs, TRF-net achieves better or comparable classification performance with much fewer parameters and sparser structures. They are also more interpretable.
BibTeX
@inproceedings{li2018building,
  author = {Xiaopeng Li and Zhourong Chen and Nevin L. Zhang},
  title = {{Building Sparse Deep Feedforward Networks using Tree Receptive Fields}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=ByFfmGdbH},
}
Figure