Component-Wise Boosting of Targets for Multi-Output Prediction

Quay Au, Daniel Schalk, Giuseppe Casalicchio, Ramona Schödel, Clemens Stachl, Bernd BischlarXiv 2019 · arXiv

Introduction

In many scientific and industrial applications, researchers aim to predict multiple related outcomes simultaneously. For instance, in psychology, behavioral data might be used to predict several personality traits and demographic factors. These targets are rarely independent; a high score in one trait often correlates with specific behaviors in another. Traditional approaches either treat these targets as independent (Independent Models, IM) or use complex "black box" stacking methods that capture dependencies but obscure the underlying logic.

To bridge this gap, researchers have introduced Component-Wise Multi-output Boosting (CMOB). This algorithm leverages a sparse boosting approach to model target dependencies explicitly, offering a rare combination of competitive accuracy and high interpretability.

The CMOB Methodology

CMOB operates within a two-level stacking framework designed to exploit correlations between targets.

Level 1: Meta-Feature Generation

The first step involves training independent models for each target. To avoid overfitting, the algorithm uses inner cross-validation (inner-CV) to generate "meta-features"—initial predictions for each target based solely on the input features.

Figure 3: Generating meta-features via inner cross-validation for each target.

Level 2: Component-Wise Boosting

In the second level, instead of a complex non-linear model, CMOB employs component-wise boosting. This step uses the initial predictions of all targets to refine the prediction of a specific target.

Figure 6: The second-level model uses the predictions of all targets to predict target $t_j$.

The "component-wise" nature is critical: in each iteration, the algorithm selects only the single most helpful target prediction to reduce the current residual error. This results in a sparse linear model where irrelevant target dependencies are automatically assigned a coefficient of zero.

Interpretability and "Self-Correction"

The sparsity of CMOB provides immediate insights into target relationships. For example, in the slump dataset (predicting concrete properties), the model revealed a "self-correction" phenomenon: the best predictor for a target's final value was often its own initial prediction from Level 1. The second level essentially performed a linear refinement of the first-level guess.

Furthermore, because the second level is an additive linear model, the coefficients can be visualized as heatmaps. This allows researchers to identify mutual influences, such as the strong positive relationship between conductivity and salinity in water quality datasets.

Performance Across Datasets

The study evaluated CMOB on 16 datasets across three categories: multi-label classification, multivariate regression, and mixed-type tasks.

  • Regression: CMOB showed significant strengths in multivariate regression, outperforming independent models and matching or exceeding standard stacking (STA) in datasets like andro, jura, and slump.
  • Classification: For multi-label classification, while CMOB provided superior interpretability, independent models remained a surprisingly robust baseline.
  • Interpretability Trade-off: Even when CMOB did not achieve the absolute highest accuracy, its ability to explain which targets influenced the final prediction made it more suitable for scientific discovery than "black box" alternatives.

Conclusion and Future Directions

CMOB offers a powerful alternative for multi-output prediction where understanding the "why" is as important as the "what." By enforcing sparsity through boosting, it reveals the structure of target dependencies. Future research will focus on scaling the method to high-dimensional target spaces (hundreds of labels) and exploring non-linear base-learners, such as splines, to capture more complex relationships without sacrificing the model's inherent clarity.

Summary updated

Abstract

Multi-output prediction deals with the prediction of several targets of possibly diverse types. One way to address this problem is the so called problem transformation method. This method is often used in multi-label learning, but can also be used for multi-output prediction due to its generality and simplicity. In this paper, we introduce an algorithm that uses the problem transformation method for multi-output prediction, while simultaneously learning the dependencies between target variables in a sparse and interpretable manner. In a first step, predictions are obtained for each target individually. Target dependencies are then learned via a component-wise boosting approach. We compare our new method with similar approaches in a benchmark using multi-label, multivariate regression and mixed-type datasets.
BibTeX
@misc{au2019component,
  author = {Quay Au and Daniel Schalk and Giuseppe Casalicchio and Ramona Schödel and Clemens Stachl and Bernd Bischl},
  title = {{Component-Wise Boosting of Targets for Multi-Output Prediction}},
  year = {2019},
  eprint = {1904.03943},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.03943},
}
Figure