ARC-LH: A New Adaptive Resampling Algorithm for Improving ANN Classifiers

Friedrich Leisch, Kurt HornikNeurIPS 1996

Introduction

Training Artificial Neural Networks (ANNs) is notoriously unstable. Because networks initialize with random weights and process training data in a random order, two networks trained on the exact same data can yield significantly different performance levels. In statistical terms, ANNs typically exhibit low bias—meaning they are powerful enough to learn complex decision boundaries—but high variance, making them highly sensitive to small changes in training conditions.

To address this, researchers have developed ensemble learning algorithms that combine multiple networks. While techniques like Bagging (Bootstrap Aggregating) help by averaging multiple networks, more strategic approaches to data sampling can yield better results. This paper introduces arc-lh, a novel ensemble learning algorithm designed to make ANNs more reliable by adaptively resampling training data to focus on the most difficult patterns.

Methodology: Adaptive Resampling and Combining

The authors aim to bridge the gap between simple resampling and more complex "arcing" (adaptive resampling and combining) methods. They compare three primary approaches to improving ANN stability:

  1. Bagging: Draws random subsets of data with replacement (using uniform probability) and trains a separate network on each subset.
  2. arc-fs (Standard Boosting): Based on the work of Freund & Schapire, this method increases the probability of selecting a data point if the previous network misclassified it. It treats error as a strict binary: 0 for correct, 1 for incorrect.
  3. arc-lh (The Proposed Method): Instead of a binary "black and white" view of misclassification, arc-lh leverages the squared output error of the network. If a network is "mostly wrong" about a pattern—even if it technically selected the correct class—that pattern's importance is increased for the subsequent training round.
Adaptive resampling and combining architecture

The final prediction is determined by a majority vote among all the trained networks. The authors also compare these ensemble methods to Jittering, a technique that adds artificial noise to the inputs to artificially expand the training set.

Experimental Setup

The study evaluates the algorithms using three well-known artificial benchmark problems designed to test complex classification boundaries:

  • 2 Spirals with noise: A 2D problem where two classes are intertwined in spiral shapes, complicated by uniform noise.
  • Continuous XOR: A 2D version of the classic XOR problem, where the classes occupy alternating quadrants of a square.
  • Ringnorm: A more complex 20-dimensional problem involving two overlapping multivariate normal distributions.

For all tests, the authors utilized Single Hidden Layer Perceptrons (SHLPs) with a sufficient number of hidden nodes to ensure the models maintained low inherent bias.

Key Findings

The core insight of the paper is the "grey-scale" advantage: for neural networks, a continuous error signal is far more informative than a binary misclassification rate. By utilizing the actual output error, arc-lh identifies patterns the network is struggling with before it outright fails on them.

The experiments confirm that the primary reason ANNs fail to reach Bayes optimal performance is not a lack of representational power (bias), but rather the erratic nature of the training process (variance). Averaging just 10 networks was sufficient to drastically reduce the variance component of the error across all tested problems.

Results comparing standard, jitter, bagging, arc-fs, and arc-lh across three datasets

As shown in the results table, arc-lh proves to be a formidable competitor to existing methods:

  • Significant Error Reduction: In the 2 Spirals task, the total risk dropped from 7.75 (standard single network) to 4.32 using arc-lh.
  • Consistency: Across all three benchmarks (Spirals, XOR, and Ringnorm), arc-lh consistently outperformed both the standard single network and the jittering method.
  • Competitive with arc-fs: arc-lh performed similarly to arc-fs, occasionally demonstrating slight advantages (e.g., achieving a risk of 3.58 versus 3.73 in the XOR task).
  • Bias vs. Variance: The empirical data clearly shows that while the "Bias" metric remained almost constant across methods, the "Variance" was often cut in half by the resampling techniques.

Limitations and Future Work

The authors are transparent that while arc-lh shows slight advantages in certain scenarios, no single algorithm (Bagging, arc-fs, or arc-lh) completely dominates the others across every single metric and dataset.

The paper concludes that further improvements are likely possible, but they will require a deeper theoretical understanding of exactly why these resampling techniques are so effective at variance reduction. The authors note they are actively investigating these theoretical properties to refine the algorithms further. Additionally, because the study focuses on artificial benchmarks, testing these methods on high-stakes, real-world datasets remains an important open area for future validation.

Summary updated

BibTeX
@inproceedings{leisch1996arc,
  author = {Friedrich Leisch and Kurt Hornik},
  title = {{ARC-LH: A New Adaptive Resampling Algorithm for Improving ANN Classifiers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1996},
}
Figure