Semi-Supervised Regression with Co-Training

Zhi-Hua Zhou, Ming LiIJCAI 2005 · OpenReview

Abstract

In many practical machine learning and data mining applications, unlabeled training examples are readily available but labeled ones are fairly expensive to obtain. Therefore, semi-supervised learning algorithms such as co-training have attracted much attention. Previous research mainly focuses on semi-supervised classification. In this paper, a co-training style semi-supervised regression algorithm, i.e. COREG, is proposed. This algorithm uses two k-nearest neighbor regressors with different distance metrics, each of which labels the unlabeled data for the other regressor where the labeling confidence is estimated through consulting the influence of the labeling of unlabeled examples on the labeled ones. Experiments show that COREG can effectively exploit unlabeled data to improve regression estimates.

Introduction

Semi-supervised learning (SSL) aims to leverage a small amount of labeled data alongside a large pool of unlabeled data to build more robust models. While SSL has seen significant success in classification tasks, its application to regression has been limited. The primary challenges in semi-supervised regression are twofold: first, estimating "confidence" for continuous outputs is difficult compared to discrete class probabilities; second, traditional co-training requires two independent "views" (feature sets) of the data, which are rarely available in practice.

The COREG Algorithm

The authors introduce COREG (CO-training REGressors), a novel approach that adapts the co-training paradigm for regression without requiring dual feature views. The methodology relies on three key innovations:

  1. Lazy Base Learners: COREG utilizes k-nearest neighbor (kNN) regressors. Because kNN is a "lazy" learner, the model can be updated instantly by adding new points to the dataset, avoiding the high computational cost of retraining complex models in every iteration.
  2. Diversity via Distance Metrics: Instead of splitting features, COREG creates functional diversity by using different Minkowski distance metrics (different $p$ values) for its two regressors. This ensures the models perceive the proximity of data points differently.
  3. Confidence through Influence: To identify which unlabeled points are "safe" to label, COREG uses a heuristic: an unlabeled point is considered confidently labeled if adding it (with its predicted value) to the training set significantly reduces the Mean Squared Error (MSE) of the regressor on the existing labeled data.

In each iteration, Regressor A identifies the unlabeled point it is most confident about and adds it to Regressor B’s training set, and vice versa. This cross-labeling mechanism prevents the models from simply reinforcing their own biases.

Experimental Results

The algorithm was evaluated on ten diverse datasets, including benchmark functions like Mexican Hat, Friedman #1-3, and SinC. The data was split into a small labeled set (10%), a large unlabeled pool (90%), and a dedicated test set (25%).

MSE reduction over iterations for various datasets

As shown in the performance curves above, COREG (represented by the black square line) consistently achieves lower MSE as iterations progress compared to the baseline (labeled-only) and other semi-supervised methods.

Quantitatively, COREG outperformed the baseline on 8 out of 10 datasets. As summarized in the table below, the gains were substantial in several cases, reaching improvements of 22% for Polynomial and 26% for SinC.

Percentage improvement of COREG compared to other methods

In contrast, alternative methods like self-training (SELF) and artificial feature splitting (ARTRE) were less consistent, sometimes even degrading performance relative to the labeled-only baseline.

Conclusion and Future Work

COREG demonstrates that co-training can be effectively applied to regression by substituting feature views with metric diversity and using MSE reduction as a proxy for confidence. While highly effective, the algorithm did struggle on the "Plane" dataset, likely due to local MSE approximations failing to reflect global error in specific distributions. Future research may explore extending this influence-based confidence measure to other base learners beyond kNN, such as Support Vector Regression or Random Forests.

Summary updated

Abstract

In many practical machine learning and data mining applications, unlabeled training examples are readily available but labeled ones are fairly expensive to obtain. Therefore, semi-supervised learning algorithms such as co-training have attracted much attention. Previous research mainly focuses on semi-supervised classification. In this paper, a co-training style semi-supervised regression algorithm, i.e. COREG, is proposed. This algorithm uses two k-nearest neighbor regressors with different distance metrics, each of which labels the unlabeled data for the other regressor where the labeling confidence is estimated through consulting the influence of the labeling of unlabeled examples on the labeled ones. Experiments show that COREG can effectively exploit unlabeled data to improve regression estimates.
BibTeX
@inproceedings{zhou2005semi,
  author = {Zhi-Hua Zhou and Ming Li},
  title = {{Semi-Supervised Regression with Co-Training}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2005},
  url = {https://openreview.net/forum?id=Sy4CMNzd-S},
}
Figure