Combining Classifiers Using Correspondence Analysis

Christopher J. MerzNeurIPS 1997

TL;DR

SCANN (Stacking, Correspondence Analysis, and Nearest Neighbor) is a novel ensemble learning method designed to combine predictions from diverse, heterogeneous classifiers. By using Correspondence Analysis to map model predictions into a geometric space, it identifies complex relationships between learners and data points, making it significantly more robust than simple voting—especially when many constituent models are poor or highly correlated.

What Problem Does This Paper Address?

In machine learning, "ensemble" methods like bagging and boosting typically improve performance by applying a single algorithm to different subsets of data. However, there is less research on how to effectively combine a collection of different types of models (e.g., a neural network, a decision tree, and a Bayesian classifier) that have different internal representations and search strategies.

The primary challenge is deciding which models to trust and how to weight them. Simple methods like Plurality Voting (PV) fail when the majority of models make the same types of errors (correlated errors). More complex "stacking" methods (where a meta-model learns to combine the base models) often overfit, reacting to insignificant fluctuations in predictions rather than meaningful patterns.

Methods

The authors introduce SCANN, which integrates three distinct components:

  1. Stacking: This provides the framework. The system first generates "out-of-sample" predictions for the training data. These predictions become the input for the next stage.
  2. Correspondence Analysis (CA): This is the core innovation. CA is a geometric technique (similar to Principal Component Analysis but for categorical data) used to explore the relationship between the rows (examples) and columns (classifier predictions) of a matrix.
Stages of Correspondence Analysis

By performing a Singular Value Decomposition (SVD) on a transformed prediction matrix, SCANN creates a new coordinate space where:

  • Examples that receive similar predictions lie close together.
  • Models that make similar predictions lie close together.
  • Examples lie close to the class labels they are most likely to belong to based on the ensemble's behavior.
  • Nearest Neighbor: To classify a new, unseen example, SCANN maps the new predictions into this established geometric space. It then uses a nearest-neighbor approach to assign the class label whose "class point" is closest to the example’s hypothesized position in the scaled space.

Datasets

The algorithm was evaluated on 18 datasets, primarily sourced from the UCI Machine Learning Database Repository. These include:

  • Standard Benchmarks: Iris, Breast Cancer, Glass, Ionosphere, Sonar, Vote, and Waveform.
  • Challenging Sets: Abalone (notorious for high error rates), Musk, and Credit.
  • Unreleased Medical Data: Two specific datasets, retardation and dementia, were included to test the algorithm on real-world clinical data.

Key Insights and Ideas

  • Geometric "Bias" Mapping: Instead of just treating model outputs as numbers, SCANN treats them as coordinates in a "bias space." This allows the algorithm to geometrically capture how different models agree or disagree on specific types of examples.
  • Robustness to "Bad" Models: A standout insight is SCANN's performance on the Abalone dataset. Even when 7 out of 8 constituent models were very poor (80% error rate) and highly correlated, SCANN successfully filtered the noise to outperform simple voting by nearly 50%.
  • Avoiding the Overfitting Trap: While other stacking methods (like using a Neural Network as a meta-learner) tend to overfit to the training predictions, SCANN’s use of CA provides a more stable representation that ignores "insignificant variations" in model predictions.

Key Results

SCANN's performance was rigorously compared against Plurality Voting (PV), Stacked Backpropagation (S-BP), Stacked Bayes (S-BAYES), and the single best individual model in the ensemble.

Performance comparison of SCANN and other methods against Plurality Voting
  • Superiority over Voting: SCANN achieved a statistically significant advantage over Plurality Voting (PV) on 7 of the 18 datasets, with error reductions ranging from 3% to 50% (indicated by bold ratios < 1.0 in the results table).
  • Consistency: Unlike other meta-learners (Stacked Backpropagation or Stacked Bayes), SCANN posted no statistically significant losses against the baseline voting method across all 18 datasets.
  • Beating the "Best" Individual: In almost every case (except for the retardation set), SCANN performed as well as or better than the single best individual model in the ensemble.
  • Direct Comparison: In head-to-head matches against other stacking techniques, SCANN won significantly in 4-5 cases and never lost.

Limitations and Future Work

  • Computational Complexity: The paper notes that CA involves SVD, which is more computationally intensive than simple voting, though it doesn't provide a detailed runtime analysis for very large-scale ensembles.
  • Dimensionality Selection: While the paper mentions that the optimal number of dimensions is determined by optimizing classification on the stacking data, the sensitivity of the model to this parameter is not explored in depth.
  • Scope: The current work focuses on classification. Extending this geometric approach to regression problems (continuous outputs) remains an open area for future research.

Summary updated

BibTeX
@inproceedings{merz1997combining,
  author = {Christopher J. Merz},
  title = {{Combining Classifiers Using Correspondence Analysis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1997},
}
Figure