Diversity Regularized Machine

Yang Yu, Yu-Feng Li, Zhi-Hua ZhouIJCAI 2011 · OpenReview

Introduction

In machine learning, ensembles—collections of models working in concert—frequently achieve higher accuracy than any individual constituent. The effectiveness of an ensemble is traditionally attributed to two factors: the accuracy of individual learners and their diversity, or the tendency to make different types of errors. Historically, diversity has been achieved through heuristic methods such as Bagging (training on random data subsets) or AdaBoost (re-weighting difficult examples). However, these approaches lack a deterministic mathematical framework to explicitly control diversity during training. The Diversity Regularized Machine (DRM) addresses this gap by treating diversity as a formal optimization constraint.

The DRM Framework

The DRM formulates the training of multiple Support Vector Machines (SVMs) as a joint optimization problem. Rather than relying on stochastic processes, it defines diversity based on the geometric relationship between the weight vectors ($w$) of linear learners. Specifically, diversity is measured by the angle between these vectors; larger angles signify greater diversity.

To implement this, a diversity constraint is integrated directly into the SVM objective function. This forces the optimization process to identify a set of models that are individually accurate yet collectively distinct. To handle the computational complexity of simultaneous optimization, the authors employ an alternating optimization technique, decomposing the global problem into manageable sub-problems solvable with standard quadratic programming tools like LIBSVM.

Diversity as Regularization

A core contribution of this work is the theoretical link between diversity and generalization. Using PAC (Probably Approximately Correct) learning theory and covering numbers, the authors prove that enforcing diversity effectively reduces the complexity of the ensemble's hypothesis space. In this light, diversity acts as a form of regularization, analogous to $L_2$ weight decay, which prevents overfitting by restricting the search space.

Experimental Evaluation

The authors evaluated DRM against 15 datasets from the UCI Machine Learning Repository, including clean1, haberman, heart, and house-votes. DRM was compared against single SVMs, Bagging, and AdaBoost.

Figure 4: Error rates for SVM and DRM across four UCI datasets as a function of the diversity parameter $\mu$.

As shown in Figure 4, the diversity-controlling parameter ($\mu$) significantly impacts performance. In many cases, there is an optimal range for $\log_{10}\mu$ where the testing error of the DRM ensemble (marked with circles) is lower than that of a standard SVM. Furthermore, the experiments demonstrate that as diversity increases, the "generalization gap" between training and testing error often shrinks, validating the theory that diversity serves as a robust regularizer that prevents overfitting.

Key Results and Insights

  • Superior Performance: DRM consistently achieved better average rankings across the 15 test datasets compared to single SVMs, Bagging, and AdaBoost.
  • Stability: Unlike AdaBoost, which can be sensitive to noise and occasionally underperform a single learner, DRM proved highly stable and "never lost" to a single SVM in the reported trials.
  • Determinism: Unlike stochastic ensemble methods, DRM is deterministic; given the same parameters, the optimization consistently yields the same diverse ensemble.

Conclusion and Future Work

The Diversity Regularized Machine moves ensemble learning from heuristic-based "black boxes" toward a mathematical science. While the current framework uses a simple averaging scheme for model combination and focuses primarily on linear classification (extended via kernels), it establishes a powerful precedent for explicit diversity control. Future research may explore more sophisticated combination strategies and the application of explicit diversity regularization to non-linear architectures such as deep neural networks.

Summary updated

BibTeX
@inproceedings{yu2011diversity,
  author = {Yang Yu and Yu-Feng Li and Zhi-Hua Zhou},
  title = {{Diversity Regularized Machine}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2011},
  url = {https://openreview.net/forum?id=fpTE3Ol3CFr},
}
Figure