Abstract
Deep neural networks often struggle in low-data regimes, suffering from extreme overfitting and training instability. This paper presents a novel ensemble approach—incorporating both "Vertical" (cross-layer) and "Horizontal" (cross-epoch) strategies—to stabilize performance and improve accuracy. Applied to the ICML 2013 Black Box Challenge, these techniques enabled a deep architecture to achieve competitive results despite having only 1,000 labeled training examples for over 16 million parameters.
Introduction
The "data-hungry" nature of deep learning models poses a significant challenge when labeled data is scarce. In the ICML 2013 Black Box Challenge, participants were tasked with training a high-capacity model on just 1,000 labeled samples, alongside 130,000 unlabeled ones. This scenario typically leads to two major hurdles: extreme overfitting, where the model memorizes the small training set, and high instability, where validation error fluctuates significantly between epochs.
Methodology
The proposed architecture utilizes five layers of Stacked Denoising Auto-encoders (SDAE) for unsupervised pre-training, followed by three Maxout layers for supervised fine-tuning. To maximize performance, the authors introduced three specific ensemble strategies.
Vertical and Horizontal Ensembling
- Vertical Voting: This method treats intermediate hidden layers as feature extractors. Separate classifiers (e.g., Random Forests) are trained on representations from different depths (h5, h6, h7) and their predictions are combined via voting.
- Horizontal Voting: To mitigate epoch-to-epoch instability, predictions are averaged across a stable range of training epochs (e.g., epochs 651 to 850).
- Horizontal Stacked Ensemble: A more advanced temporal ensemble where softmax outputs from multiple epochs are concatenated and fed into a meta-classifier (Random Forest) for final prediction.
Dataset and Experimental Setup
The study uses the ICML 2013 Black Box Challenge dataset, consisting of 1,875-dimensional anonymized features across 9 classes. The setup includes 130,000 unlabeled samples for pre-training and a mere 1,000 labeled samples for training. Evaluation is performed on a 10,000-sample test set split into public and private leaderboards.
Results and Discussion
The integration of unsupervised pre-training and ensemble techniques yielded substantial improvements. The jump from Model 1 (no pre-training) to Model 2 (with pre-training) highlights the necessity of leveraging unlabeled data in low-resource settings.
As shown in the results, the Horizontal Stacked Ensemble (Model 6) consistently outperformed the base deep model (Model 2), reaching 68.54% accuracy on the public leaderboard. This approach secured top-tier placements in the competition (3rd/4th public, 7th/5th private).
Key Insights
- Temporal Regularization: Averaging predictions across epochs (Horizontal) acts as a regularizer against the "fluke" weights of any single epoch.
- Stacking vs. Voting: Learning how to combine predictions via a meta-classifier (Stacking) proved superior to simple averaging.
- Feature Diversity: While Vertical Voting explores hierarchical representations, the authors noted it can be inconsistent, suggesting that layers within a single network may lack sufficient diversity for optimal ensembling.
Conclusion
By combining unsupervised pre-training with temporal and hierarchical ensemble methods, this work demonstrates a robust pathway for applying deep learning to small-scale labeled datasets. Future research may explore more sophisticated weighting strategies for layer-wise and epoch-wise contributions to further enhance stability and accuracy.
Summary updated