Abstract
In many practical machine learning and data mining applications, unlabeled training examples are readily available but labeled ones are fairly expensive to obtain. Therefore, semi-supervised learning algorithms such as co-training have attracted much attention. Previous research mainly focuses on semi-supervised classification. In this paper, a co-training style semi-supervised regression algorithm, i.e. COREG, is proposed. This algorithm uses two k-nearest neighbor regressors with different distance metrics, each of which labels the unlabeled data for the other regressor where the labeling confidence is estimated through consulting the influence of the labeling of unlabeled examples on the labeled ones. Experiments show that COREG can effectively exploit unlabeled data to improve regression estimates.
Introduction
Semi-supervised learning (SSL) aims to leverage a small amount of labeled data alongside a large pool of unlabeled data to build more robust models. While SSL has seen significant success in classification tasks, its application to regression has been limited. The primary challenges in semi-supervised regression are twofold: first, estimating "confidence" for continuous outputs is difficult compared to discrete class probabilities; second, traditional co-training requires two independent "views" (feature sets) of the data, which are rarely available in practice.
The COREG Algorithm
The authors introduce COREG (CO-training REGressors), a novel approach that adapts the co-training paradigm for regression without requiring dual feature views. The methodology relies on three key innovations:
- Lazy Base Learners: COREG utilizes k-nearest neighbor (kNN) regressors. Because kNN is a "lazy" learner, the model can be updated instantly by adding new points to the dataset, avoiding the high computational cost of retraining complex models in every iteration.
- Diversity via Distance Metrics: Instead of splitting features, COREG creates functional diversity by using different Minkowski distance metrics (different $p$ values) for its two regressors. This ensures the models perceive the proximity of data points differently.
- Confidence through Influence: To identify which unlabeled points are "safe" to label, COREG uses a heuristic: an unlabeled point is considered confidently labeled if adding it (with its predicted value) to the training set significantly reduces the Mean Squared Error (MSE) of the regressor on the existing labeled data.
In each iteration, Regressor A identifies the unlabeled point it is most confident about and adds it to Regressor B’s training set, and vice versa. This cross-labeling mechanism prevents the models from simply reinforcing their own biases.
Experimental Results
The algorithm was evaluated on ten diverse datasets, including benchmark functions like Mexican Hat, Friedman #1-3, and SinC. The data was split into a small labeled set (10%), a large unlabeled pool (90%), and a dedicated test set (25%).
As shown in the performance curves above, COREG (represented by the black square line) consistently achieves lower MSE as iterations progress compared to the baseline (labeled-only) and other semi-supervised methods.
Quantitatively, COREG outperformed the baseline on 8 out of 10 datasets. As summarized in the table below, the gains were substantial in several cases, reaching improvements of 22% for Polynomial and 26% for SinC.
In contrast, alternative methods like self-training (SELF) and artificial feature splitting (ARTRE) were less consistent, sometimes even degrading performance relative to the labeled-only baseline.
Conclusion and Future Work
COREG demonstrates that co-training can be effectively applied to regression by substituting feature views with metric diversity and using MSE reduction as a proxy for confidence. While highly effective, the algorithm did struggle on the "Plane" dataset, likely due to local MSE approximations failing to reflect global error in specific distributions. Future research may explore extending this influence-based confidence measure to other base learners beyond kNN, such as Support Vector Regression or Random Forests.
Summary updated