TL;DR
This paper tackles two key challenges in stacked generalization for classification: choosing the right "level-1" model and the right input features. They demonstrate that using class probabilities (confidence) from base models as input to a multi-response linear regression (MLR) level-1 model significantly improves accuracy over using just class predictions.
What Problem Does This Paper Address?
Stacked generalization is a powerful technique for combining multiple machine learning models to achieve better predictive accuracy than any single model. It works by training "level-0" models on the data, then using their predictions as input features to train a "level-1" model. However, when applied to classification tasks, two aspects were considered "black art" and poorly understood:
- What kind of "level-1" generalizer (model) should be used?
- What features should be fed into this level-1 generalizer?
Previous work, particularly in regression, showed success with specific constraints (like non-negativity of coefficients) for the level-1 model. This paper aims to clarify these issues for classification, building on prior work by Breiman and others.
Methods
The core of the paper is an empirical investigation into stacked generalization for classification.
- Level-0 Generalizers: They used three common algorithms: C4.5 (decision trees), Naive Bayes (NB), and IB1 (a k-nearest neighbor variant). For these, they focused on obtaining class probabilities (confidence estimates) rather than just the single predicted class.
- Level-1 Generalizers: They experimented with four algorithms: C4.5, IB1, NB, and a Multi-Response Linear Regression (MLR). The MLR approach is an adaptation of Breiman's method for regression, where each class is treated as a separate regression problem.
- Key Innovation: The paper's main contribution lies in comparing two ways of forming the level-1 data:
- Model M: Using the predicted class labels from level-0 models as input.
- Model M': Using the class probabilities (confidence scores) from level-0 models as input.
- Non-negativity Constraints: They also investigated whether non-negativity constraints on MLR coefficients, found crucial in regression, are necessary for classification.
Datasets
The authors used a mix of artificial and real-world datasets:
- Artificial Datasets: Led24, Waveform.
- Real-World Datasets (from UCI Repository): Horse, Credit, Vowel, Euthyroid, Splice, Abalone, Nettalk(s), Coding.
These datasets vary in size, number of classes, and attribute types (nominal, binary, continuous).
Key Insights and Ideas
- Confidence is King: Using class probabilities (confidence) from level-0 models as input to the level-1 model (resulting in M') is crucial for successful stacked generalization in classification. Simply using predicted class labels (M) performs poorly.
- MLR Shines at Level-1: Among the tested level-1 generalizers, Multi-Response Linear Regression (MLR) consistently outperformed C4.5, NB, and IB1. This suggests MLR's linear combination approach is well-suited for pooling confidence measures.
- Non-negativity Constraints are Unnecessary (for accuracy): Unlike in regression, non-negativity constraints on MLR coefficients did not improve predictive accuracy in classification tasks. However, they do improve interpretability.
- Interpretability of MLR: The weights learned by MLR provide insights into how much each level-0 model contributes to the prediction of each class, making the overall model more understandable.
- Robustness to Level-0 Performance: Stacked generalization, particularly with MLR, can effectively combine models even when some level-0 generalizers perform significantly worse than others.
Key Results
- M' vs. M: Model M' (using probabilities) achieved significantly better error rates than Model M (using predictions) across most datasets.
- MLR Dominance: MLR as the level-1 generalizer for M' was the best performing approach, outperforming even the best single level-0 model (selected by cross-validation, termed "BestCV") on many datasets.
- Comparison to Majority Vote, Arcing, Bagging: Stacked generalization with MLR (M') was competitive with, and often superior to, majority vote, arcing, and bagging, especially on larger datasets. MLR achieved 8 wins out of 10 comparisons against majority vote.
- Non-negativity Constraint Impact: Error rates were nearly identical with or without non-negativity constraints for MLR. However, non-negative weights led to more interpretable models.
Limitations and Future Work
- MLR Limitations: The paper acknowledges the known limitations of MLR, such as its tendency to create convex, singly connected decision regions and its suitability for unimodal probability densities. This suggests that MLR might not be optimal for all types of classification problems.
- Dataset Size: The authors note that cross-validation can produce poor estimates for very small datasets, impacting the reliability of results for those cases.
- Alternative Level-1 Generalizers: While MLR was the best found, the paper suggests exploring other algorithms like neural networks for the level-1 generalizer, though they found neural networks to be significantly slower.
- Interpretability vs. Accuracy: The trade-off between interpretability (enhanced by non-negativity constraints) and slight potential performance differences is highlighted.
Summary updated