Introduction
This paper introduces Pseudo Multi-view Co-training (PMC), a method that brings the power of co-training to standard "single-view" datasets. By automatically splitting a single feature set into two distinct, non-overlapping "pseudo-views" that satisfy theoretical expansion properties, the authors allow classifiers to "teach each other" even when the data doesn't naturally come from two different sources.
The Challenge of Single-View Data
Co-training is a highly effective semi-supervised learning technique where two classifiers are trained on two different "views" of the same data (e.g., the text of a webpage vs. the anchor text of links pointing to it). They iteratively label data for each other, bootstrapping a small amount of labeled data into a much larger set.
However, co-training has a major catch: it traditionally requires multi-view data—two sets of features that are both sufficient for classification and conditionally independent. Most real-world data, such as a simple collection of images, is "single-view." Previous attempts to force co-training onto single-view data relied on random feature splits, which often fail because they don't satisfy the underlying mathematical requirements for co-training to succeed.
Methodology: Pseudo Multi-view Decomposition
The authors propose a novel algorithm called Pseudo Multi-view Decomposition (PMD) to create these views automatically. The core of their approach is a single optimization problem that balances three goals:
- Accuracy: Both classifiers must perform well on the initial labeled data (using a softmax-relaxed loss function).
- Feature Disjointness: A hard constraint ($\sum u_i^2 v_i^2 = 0$) ensures that if one classifier uses a specific feature, the other cannot. They are forced to look at different "parts" of the data.
- $\epsilon$-Expandability: Inspired by learning theory (Balcan et al., 2004), the authors add a constraint that forces the two classifiers to be confident about different subsets of the unlabeled data. This ensures that when one classifier labels a new example, it actually provides "new" information to the other.
For multiclass problems, they employ Group Lasso regularization ($\ell_{2,1}$ norm) to ensure that the feature split remains consistent across all possible classes.
Experiments and Key Insights
The researchers evaluated PMC on two primary benchmarks: a synthetic paired digits dataset (USPS) and the challenging Caltech-256 object recognition task.
The "Aha" Moment in Visualization
In the USPS paired digits experiment, the algorithm was given a jumble of pixels representing two different digits glued together. Without being told how to separate them, the algorithm successfully learned to ignore the right digit to form one view and ignore the left digit to form the other, perfectly recovering the "natural" split.
By treating the feature split as part of the optimization problem rather than relying on random guesses, the algorithm "finds" the views that make co-training work. This led to a 3.99% error rate, vastly outperforming the baseline (18.64%) and random feature splitting (13.78%).
Noise Filtering in the Wild
To test the "weakly-supervised" potential on Caltech-256, the authors supplemented a small labeled set with 300 "noisy" images per class retrieved directly from Bing Image Search. PMC proved exceptionally good at "cherry-picking" high-quality examples from these noisy results.
Because PMC only adds examples where one classifier is highly confident and the other is not, it naturally ignores irrelevant search results (like an image of the Eiffel Tower when searching for "boxing gloves"). In this weakly-supervised setting with only 5 labeled images per class, PMC reached 29.2% accuracy, surpassing the previous best-published result of 27.1%.
Limitations and Future Work
- Computational Cost: Training on a large-scale dataset (80,000 images) took roughly 12 hours. While the testing phase is nearly instantaneous, the initial training is computationally intensive.
- Linearity: The current model uses linear logistic regression. Extending this to non-linear kernels or deep architectures could be a promising future direction.
- Domain Specificity: While tested on images, the authors suggest the method could be highly effective for other "noisy" domains like web-spam classification or sentiment analysis where feature views aren't obvious.
Summary updated