Multi-kernel maximum entropy discrimination for multi-view learning

Guoqing Chao, Shiliang SunIntell. Data Anal 2016 · DOI

TL;DR

This paper introduces Multi-Kernel Maximum Entropy Discrimination (MKMED), a novel approach to multi-view learning (MVL) that leverages the complementary nature of different data views. By combining multiple kernel functions to represent each view and then applying the Maximum Entropy Discrimination (MED) framework, MKMED effectively integrates information from diverse sources, outperforming existing single-view and multi-view methods on several real-world datasets.

What Problem Does This Paper Address?

In many real-world applications, data can be represented in multiple ways, or "views." For instance, a web page can be described by its text content and by the text of links pointing to it. Multi-view learning (MVL) aims to harness the power of these multiple representations to improve learning performance. A key challenge in MVL is effectively combining information from these different views, which may contain complementary insights. Existing MVL methods often focus on either consensus (agreement between views) or complementarity (unique information in each view). This paper addresses the gap in effectively utilizing the complementary information across multiple views within a unified discriminative learning framework.

Methods

The core contribution of this paper is the Multi-Kernel Maximum Entropy Discrimination (MKMED) method. MKMED builds upon the Maximum Entropy Discrimination (MED) framework, which integrates maximum entropy and maximum margin principles for discriminative estimation.

Here's a breakdown of the key methodological aspects:

  • Multi-Kernel Learning (MKL): MKMED utilizes MKL to represent each view. Instead of using a single kernel per view, it allows for the use of multiple kernels. These kernels are then combined using a weighted sum, where the weights (e.g., $\beta$ and $1-\beta$ for two views) are learned or set to balance the contribution of each view. This kernel combination is designed to capture the complementary information present in different views.
  • MED Framework: The combined kernel representation is then fed into the MED framework. MED is a general framework for discriminative estimation that aims to find a distribution over model parameters and margins, incorporating both prior knowledge and margin constraints. The paper adapts the MED formulation to handle multi-view data by using the combined kernel.
  • Complementary Principle: Unlike some other MVL methods that focus on consensus, MKMED explicitly leverages the complementary principle. This means it assumes that each view provides unique information that, when combined, leads to a more robust and accurate model.
  • Instantiation: The paper provides a specific instantiation of MKMED, detailing the prior distributions and the optimization problem. This involves deriving the dual optimization problem and then replacing the dot product with the combined kernel function.

Datasets

The proposed MKMED method was evaluated on three real-world datasets:

  • Web-page dataset: This dataset consists of web pages, with two views representing words on the page and words in the links pointing to it. The task is to predict if a web page is a course home page.
  • Ionosphere dataset: This dataset, originally a single-view dataset, was augmented by generating a second view using Principal Component Analysis (PCA). The task involves classifying radar returns from the ionosphere.
  • Advertisement dataset: This dataset contains advertisements, with two views derived from image-related features (URL, alt text, caption) and other features (words from referring page URLs). The task is to classify images as advertisements or non-advertisements.

Key Insights and Ideas

  • Complementarity is Key: The paper strongly advocates for the effectiveness of the complementary principle in MVL. By explicitly designing the method to exploit unique information in each view, MKMED demonstrates superior performance.
  • MKL as a Unifier: Multi-kernel learning provides a flexible way to encode diverse notions of similarity within each view. Combining these kernels allows for a richer representation that can be effectively used by discriminative models like MED.
  • MED's Discriminative Power: The MED framework, with its emphasis on maximum entropy and maximum margin, proves to be a suitable foundation for integrating multi-view information in a discriminative manner.
  • Flexibility in Kernel Combination: The weighted kernel combination allows for tuning the influence of each view, offering a degree of flexibility in how complementary information is integrated.

Key Results

The experimental results on the three datasets demonstrate the effectiveness of MKMED:

  • Outperforms Single-View Methods: MKMED consistently outperforms single-view versions (MKMED1, MKMED2) and a concatenated view approach (MKMED3), highlighting the benefits of MVL.
  • Competitive with State-of-the-Art: MKMED shows comparative performance with the state-of-the-art multi-view method MVMED, and in some cases, it even exceeds MVMED's performance.
  • Superior to SVM-2K: MKMED generally outperforms SVM-2K, another competing MVL method.
  • Specific Dataset Performance: On the web-page dataset, MKMED achieved the best performance. On the ionosphere dataset, MKMED, MVMED, and MKMED3 performed best. For the advertisement dataset, MVMED was slightly better, but MKMED was very competitive and sometimes surpassed it.

Limitations and Future Work

  • Computational Complexity: The paper acknowledges that the computational complexity of MKMED is $O(l^3)$, similar to standard SVM, which can be a challenge for large-scale applications.
  • Future Work: The authors suggest several promising directions:
  • Nonlinear Kernel Combinations: Exploring more sophisticated nonlinear kernel combination methods beyond simple weighted sums.
  • Speedup Strategies: Investigating techniques like stochastic optimization and Nyström approximation to make MKMED scalable for large datasets.
  • Extension to More Views: While the paper focuses on two views for simplicity, extending the framework to handle more than three views efficiently is a natural next step.

Summary updated

BibTeX
@article{chao2016multi,
  author = {Guoqing Chao and Shiliang Sun},
  title = {{Multi-kernel maximum entropy discrimination for multi-view learning}},
  journal = {Intell. Data Anal.},
  year = {2016},
  doi = {10.3233/ida-160816},
}
Figure