Horizontal and Vertical Ensemble with Deep Representation for Classification

Jingjing Xie, Bing Xu, Chuang ZhangarXiv 2013 · arXiv

Abstract

Deep neural networks often struggle in low-data regimes, suffering from extreme overfitting and training instability. This paper presents a novel ensemble approach—incorporating both "Vertical" (cross-layer) and "Horizontal" (cross-epoch) strategies—to stabilize performance and improve accuracy. Applied to the ICML 2013 Black Box Challenge, these techniques enabled a deep architecture to achieve competitive results despite having only 1,000 labeled training examples for over 16 million parameters.

Introduction

The "data-hungry" nature of deep learning models poses a significant challenge when labeled data is scarce. In the ICML 2013 Black Box Challenge, participants were tasked with training a high-capacity model on just 1,000 labeled samples, alongside 130,000 unlabeled ones. This scenario typically leads to two major hurdles: extreme overfitting, where the model memorizes the small training set, and high instability, where validation error fluctuates significantly between epochs.

Methodology

The proposed architecture utilizes five layers of Stacked Denoising Auto-encoders (SDAE) for unsupervised pre-training, followed by three Maxout layers for supervised fine-tuning. To maximize performance, the authors introduced three specific ensemble strategies.

The deep architecture featuring SDAE pre-training layers (h0-h4) and Maxout layers (h5-h7).

Vertical and Horizontal Ensembling

  1. Vertical Voting: This method treats intermediate hidden layers as feature extractors. Separate classifiers (e.g., Random Forests) are trained on representations from different depths (h5, h6, h7) and their predictions are combined via voting.
  2. Horizontal Voting: To mitigate epoch-to-epoch instability, predictions are averaged across a stable range of training epochs (e.g., epochs 651 to 850).
  3. Horizontal Stacked Ensemble: A more advanced temporal ensemble where softmax outputs from multiple epochs are concatenated and fed into a meta-classifier (Random Forest) for final prediction.

Dataset and Experimental Setup

The study uses the ICML 2013 Black Box Challenge dataset, consisting of 1,875-dimensional anonymized features across 9 classes. The setup includes 130,000 unlabeled samples for pre-training and a mere 1,000 labeled samples for training. Evaluation is performed on a 10,000-sample test set split into public and private leaderboards.

Results and Discussion

The integration of unsupervised pre-training and ensemble techniques yielded substantial improvements. The jump from Model 1 (no pre-training) to Model 2 (with pre-training) highlights the necessity of leveraging unlabeled data in low-resource settings.

Comparison of accuracy across different model configurations on public and private test sets.

As shown in the results, the Horizontal Stacked Ensemble (Model 6) consistently outperformed the base deep model (Model 2), reaching 68.54% accuracy on the public leaderboard. This approach secured top-tier placements in the competition (3rd/4th public, 7th/5th private).

Key Insights

  • Temporal Regularization: Averaging predictions across epochs (Horizontal) acts as a regularizer against the "fluke" weights of any single epoch.
  • Stacking vs. Voting: Learning how to combine predictions via a meta-classifier (Stacking) proved superior to simple averaging.
  • Feature Diversity: While Vertical Voting explores hierarchical representations, the authors noted it can be inconsistent, suggesting that layers within a single network may lack sufficient diversity for optimal ensembling.

Conclusion

By combining unsupervised pre-training with temporal and hierarchical ensemble methods, this work demonstrates a robust pathway for applying deep learning to small-scale labeled datasets. Future research may explore more sophisticated weighting strategies for layer-wise and epoch-wise contributions to further enhance stability and accuracy.

Summary updated

BibTeX
@misc{xie2013horizontal,
  author = {Jingjing Xie and Bing Xu and Chuang Zhang},
  title = {{Horizontal and Vertical Ensemble with Deep Representation for Classification}},
  year = {2013},
  eprint = {1306.2759},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1306.2759},
}
Figure