Soft Margin Consistency Based Scalable Multi-View Maximum Entropy Discrimination

Liang Mao, Shiliang SunIJCAI 2016 · OpenReview

Abstract

Multi-view learning receives increasing interest in recent years to analyze complex data. Lately, multiview maximum entropy discrimination (MVMED) and alternative MVMED (AMVMED) were proposed as extensions of maximum entropy discrimination (MED) to the multi-view learning setting, which use the hard margin consistency principle that enforces two view margins to be the same. In this paper, we propose soft margin consistency based multi-view MED (SMVMED) achieving margin consistency in a less strict way, which minimizes the relative entropy between the posteriors of two view margins. With a trade-off parameter balancing large margin and margin consistency, SMVMED is more flexible. We also propose a sequential minimal optimization (SMO) algorithm to efficiently train SMVMED and make it scalable to large datasets. We evaluate the performance of SMVMED on multiple real-world datasets and get encouraging results.

Introduction

In modern machine learning, "multi-view" data refers to objects described by distinct sets of features. A classic example is a webpage, which can be represented by its internal text content (View 1) and the anchor text of inbound hyperlinks (View 2). Multi-view learning aims to leverage these complementary information sources by training classifiers jointly, often leading to better generalization than single-view approaches.

Previous state-of-the-art methods, such as Multi-View Maximum Entropy Discrimination (MVMED) and its alternative variant (AMVMED), were built on a hard margin consistency principle. This principle mandates that the classification margins (the "confidence" of the classifier) must be identical across all views. While theoretically sound, this requirement is often too restrictive for real-world data where one view might be significantly noisier or more ambiguous than another. Furthermore, the computational complexity of these models prevented them from scaling to large-scale datasets.

The SMVMED Framework

To address these limitations, this paper introduces SMVMED (Soft Margin Consistency Based Multi-View Maximum Entropy Discrimination). The framework incorporates three primary innovations:

  1. Soft Margin Consistency: Instead of enforcing identical margins, SMVMED minimizes the Kullback-Leibler (KL) divergence between the posterior distributions of the margins from different views. This "soft" agreement acts as a regularizer, allowing for view-specific nuances while ensuring overall consensus. A trade-off parameter ($\alpha$) allows practitioners to balance individual view performance with cross-view consistency.
  2. Gaussian Margin Priors: Traditional MED models typically utilize exponential priors, which result in complex logarithmic terms during optimization. By adopting Gaussian priors, the authors transform the dual problem into a standard Quadratic Programming (QP) problem, which is much more amenable to efficient solvers.
  3. Scalable Optimization via SMO: To ensure the model can handle "big data" scenarios, the authors developed a specialized Sequential Minimal Optimization (SMO) algorithm. By decomposing the large QP problem into the smallest possible sub-problems that can be solved analytically, SMVMED achieves significant speedups and can process datasets with tens of thousands of examples.

Experimental Evaluation

The researchers evaluated SMVMED on three real-world benchmarks: the Course dataset (WebKB), the Ads dataset (internet advertisements), and the Indoor dataset (UJIIndoorLoc WiFi fingerprints).

Accuracy and Efficiency

As shown in the results, the "soft" approach provides a clear performance boost. SMVMED consistently outperformed hard-margin versions (MVMED and AMVMED) across the benchmarks.

Table 1: Accuracy comparison on Course and Ads datasets. SMVMED achieves the highest accuracy by leveraging soft margin consistency.

The efficiency gains are even more pronounced. The transition to Gaussian priors and the implementation of the SMO algorithm resulted in massive speedups. On the Course dataset, SMVMED reduced training time from over 260 seconds to just 1.41 seconds.

Table 2: Training time comparison. The combination of Gaussian priors and SMO allows SMVMED to train orders of magnitude faster than exponential-prior variants.

Scalability

On the large-scale Indoor dataset, which was previously inaccessible to MVMED methods due to its size, SMVMED achieved 100% accuracy. It also demonstrated superior scalability compared to other multi-view baselines like SVM-2K, running nearly 7 times faster (448s vs 3113s).

Conclusion

SMVMED represents a significant step forward in multi-view learning. By relaxing the rigid hard-margin constraint and optimizing the underlying mathematical formulation for speed, the authors have created a framework that is both more accurate and significantly more practical for large-scale applications. Future work may involve exploring non-linear kernel selections, refining the selection of the $\alpha$ parameter, and extending the soft-consistency mechanism to handle more than two views.

Summary updated

Abstract

Multi-view learning receives increasing interest in recent years to analyze complex data. Lately, multiview maximum entropy discrimination (MVMED) and alternative MVMED (AMVMED) were proposed as extensions of maximum entropy discrimination (MED) to the multi-view learning setting, which use the hard margin consistency principle that enforces two view margins to be the same. In this paper, we propose soft margin consistency based multi-view MED (SMVMED) achieving margin consistency in a less strict way, which minimizes the relative entropy between the posteriors of two view margins. With a trade-off parameter balancing large margin and margin consistency, SMVMED is more flexible. We also propose a sequential minimal optimization (SMO) algorithm to efficiently train SMVMED and make it scalable to large datasets. We evaluate the performance of SMVMED on multiple real-world datasets and get encouraging results.
BibTeX
@inproceedings{mao2016soft,
  author = {Liang Mao and Shiliang Sun},
  title = {{Soft Margin Consistency Based Scalable Multi-View Maximum Entropy Discrimination}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
  url = {https://openreview.net/forum?id=Hk4htmfd-S},
}
Figure