Learning using privileged information: SV M+ and weighted SVM

Maksim Lapin, Matthias Hein, Bernt Schiele

Abstract

This paper explores the theoretical and empirical relationship between the Learning Using Privileged Information (LUPI) paradigm and the standard Weighted SVM (WSVM). The authors demonstrate that the sophisticated SVM+ algorithm is a special case of WSVM, where privileged information acts as a mechanism for determining instance weights. This insight allows for the development of algorithms that can learn optimal weights even in the absence of privileged features, significantly improving classification performance.

Introduction

In many machine learning scenarios, "privileged information" is available during training that will not be accessible at test time. For example, a medical diagnosis model might be trained using expensive biopsy results, but must eventually function using only non-invasive ultrasound images. Vapnik’s LUPI paradigm introduced SVM+ to handle this, but the algorithm's mathematical complexity has often obscured its relationship to more traditional methods. This paper addresses whether SVM+ can be simplified and if its benefits can be achieved without explicit privileged features.

The Equivalence of SVM+ and Weighted SVM

Using convex analysis and Karush-Kuhn-Tucker (KKT) conditions, the researchers prove a fundamental mapping: for any SVM+ solution, there exists a set of instance weights that allows a standard Weighted SVM to produce the exact same classifier. The weights are derived directly from the dual variables of the SVM+ optimization problem.

This equivalence reveals that the "secret sauce" of SVM+ is data re-weighting. Privileged features essentially serve to estimate the loss (slack variables) for each training point, forcing the model to focus on "hard" examples.

Figure 1: Illustration of the relationship between privileged space features and instance weights in the input space.

As shown in Figure 1, there is a clear correlation between the privileged space $\tilde{X}$ and the instance weights $c$. In this scenario, the privileged information helps identify difficult points, leading to a positive correlation ($\rho = 0.33$) between weights and the estimated slack $\xi^*$. This re-weighting guides the decision boundary more effectively than a standard unweighted SVM.

Flexibility and Weight Learning

While SVM+ is powerful, the authors identify a specific bias: it is mathematically constrained to put more weight on examples with higher loss. In contrast, WSVM is more general because it can also choose to down-weight outliers—a capability SVM+ lacks.

Building on this, the authors propose a method to learn these weights directly using a validation set and gradient descent (specifically the BFGS algorithm). This allows the model to achieve LUPI-like performance boosts even when no privileged features are available, simply by identifying which training points are most informative for generalization.

Experimental Results

The authors validated their findings across several benchmarks:

  • MNIST Digits: Successfully replicated Vapnik’s original results using the Weighted SVM framework, confirming the theoretical equivalence.
  • Spambase Dataset: The weight-learning approach achieved an error rate comparable to a standard SVM trained on twice as much data.
  • Outlier Robustness: In toy datasets, WSVM recovered near-optimal decision boundaries by assigning zero weight to extreme outliers, whereas standard SVMs were heavily skewed.
  • Human Intuition: Using human "confidence scores" as privileged information improved performance on small training sets, demonstrating that subjective human knowledge can be effectively encoded as instance weights.

Conclusion

The study concludes that SVM+ is a subset of the more flexible Weighted SVM framework. By reframing privileged information as a weighting mechanism, the authors provide a simpler, more general tool for improving classifier performance. Their weight-learning algorithm demonstrates that the benefits of LUPI can be harvested even without "hints," provided a small validation set is available to guide the weighting process.

Summary updated

Abstract

Prior knowledge can be used to improve predictive performance of learning algorithms or reduce the amount of data required for training. The same goal is pursued within the learning using privileged information paradigm which was recently introduced by Vapnik et al. and is aimed at utilizing additional information available only at training time -- a framework implemented by SVM+. We relate the privileged information to importance weighting and show that the prior knowledge expressible with privileged features can also be encoded by weights associated with every training example. We show that a weighted SVM can always replicate an SVM+ solution, while the converse is not true and we construct a counterexample highlighting the limitations of SVM+. Finally, we touch on the problem of choosing weights for weighted SVMs when privileged features are not available.
BibTeX
@article{lapin2014learning,
  author = {Maksim Lapin and Matthias Hein and Bernt Schiele},
  title = {{Learning using privileged information: SV M+ and weighted SVM}},
  journal = {Neural Networks},
  year = {2014},
  doi = {10.1016/j.neunet.2014.02.002},
}
Figure