Combining Gradient Boosting Machines with Collective Inference to Predict Continuous Values

Iman Al-Odah, Jennifer NevillearXiv 2016 · arXiv

Introduction

Traditional supervised learning methods, such as standard Gradient Boosting (GB) machines, typically operate under the assumption that data points are independent and identically distributed (i.i.d.). However, real-world datasets often exhibit relational structures—such as social networks or citation graphs—where linked instances are highly correlated. While Collective Inference (CI) has been successfully employed to exploit these network links for discrete classification tasks, its application to regression (predicting continuous values) has remained relatively unexplored. This paper introduces a Modified Gradient Boosting (MGB) algorithm designed to bridge this gap by integrating CI directly into the boosting process.

Methodology

The proposed Modified Gradient Boosting (MGB) algorithm adapts the stage-wise additive modeling of standard gradient boosting for relational data. In standard GB, each new weak learner (typically a regression tree) is trained to predict the residuals of the preceding ensemble. MGB enhances this by embedding an Iterative Classification Algorithm adapted for continuous values (ICA2) into the learning phase.

The Modified Gradient Boosting (MGB) algorithm integrates collective inference and relational feature engineering into the stage-wise boosting process.

The MGB process involves several key steps:

  1. Base Model Initialization: A relational regression tree is learned, and initial continuous target values for unlabeled nodes are collectively inferred. The model uses the median in terminal nodes to mitigate prediction skew.
  2. Iterative Residual Boosting: At each stage, residuals are computed, relational features are updated, and a new tree is fitted to these residuals.
  3. Relational Feature Engineering: Four specific relational features ($Rf$) are aggregated from a node's neighbors:
  4. $Rf_1$: Median of neighbors' target values.
  5. $Rf_2$: Median of neighbors' residuals at the current stage.
  6. $Rf_3$: Median of neighbors' residuals from the previous stage.
  7. $Rf_4$: The instance's target value as estimated by the current model.

Experimental Evaluation

The algorithm was evaluated using the IMDB dataset, consisting of 10,770 movies interconnected by 212,460 edges (shared producers). The objective was to predict a movie's total revenue, a continuous target variable. Each movie was characterized by 29 non-relational genre features and the four engineered relational features.

Key Results and Discussion

MGB consistently outperformed both standard Gradient Boosting (GB) and a simple Relational Gradient Boosting (RGB) baseline—which includes relational features but lacks collective inference—across various training set proportions.

Comparison of Root Mean Square Error (RMSE) across different training set proportions for standard GB, Relational GB (RGB), and Modified GB (MGB).

As shown in the results, MGB maintains a lower RMSE even as the proportion of labeled data increases. While the performance gap narrows as the network becomes more fully labeled, the advantage of collective inference remains significant.

Ablation studies revealed that not all relational features contribute equally. Features leveraging the target label ($Rf_1$) and current predictions ($Rf_4$) were highly effective. Conversely, features relying solely on neighbors' residuals ($Rf_2, Rf_3$) were found to be difficult to estimate via CI and could occasionally degrade performance.

Conclusion

This work demonstrates that collective inference is a viable and effective technique for regression in relational networks when integrated with gradient boosting. Future research will focus on improving the initialization of relational features and exploring more robust methods for residual estimation across network links to further enhance predictive accuracy.

Summary updated

BibTeX
@misc{alodah2016combining,
  author = {Iman Al-Odah and Jennifer Neville},
  title = {{Combining Gradient Boosting Machines with Collective Inference to Predict Continuous Values}},
  year = {2016},
  eprint = {1607.00110},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1607.00110},
}
Figure