Multi-View Local Learning

Dan Zhang, Fei Wang, Changshui Zhang, Tao LiAAAI 2008 · OpenReview

Abstract

The idea of local learning, i.e., classifying a particular example based on its neighbors, has been successfully applied to many semi-supervised and clustering problems recently. However, the local learning methods developed so far are all devised for single-view problems. In fact, in many real-world applications, examples are represented by multiple sets of features. In this paper, we extend the idea of local learning to multi-view problem, design a multi-view local model for each example, and propose a Multi-View Local Learning Regularization (MVLL-Reg) matrix. Both its linear and kernel version are given. Experiments are conducted to demonstrate the superiority of the proposed method over several state-of-the-art ones.

Introduction

In modern machine learning, "multi-view" data—where objects are described by multiple independent feature sets—is ubiquitous. For instance, a webpage can be characterized by its textual content (View 1) and its hyperlink structure (View 2). Traditional approaches often attempt to solve multi-view problems by globally combining views, such as averaging kernels or graph Laplacians across the entire dataset.

However, global models can be compromised by distant data points that do not share local characteristics. Following Vapnik's principle that it is often easier to find a good predictor for a small, local neighborhood than for the entire input space, this paper introduces a dedicated local learning framework for multi-view data: Multi-View Local Learning Regularization (MVLL-Reg).

The MVLL-Reg Framework

The core of the MVLL-Reg approach is the construction of a regularization matrix that captures local structures across multiple views. The process involves two primary steps:

  1. Defining the Local Region: For any given data point, its neighborhood is defined as the union of its nearest neighbors in each individual view. This ensures that if a point is proximal to another in any feature space, they are treated as neighbors.
  2. The Local Model: For each point, a local regressor is trained with a three-fold optimization goal: minimizing error on neighbors in View 1, minimizing error on neighbors in View 2, and enforcing an "agreement" term that requires predictions from both views to be consistent for that specific central point.

These local optimizations are integrated into a global regularization matrix ($L_{MVLL}$), which can be utilized in standard semi-supervised learning or spectral clustering frameworks. The paper provides both linear and kernel versions to handle varying data complexities.

Experimental Evaluation

The researchers evaluated MVLL-Reg on two benchmark document-mining datasets:

  • WebKB: Webpages from four universities (Cornell, Washington, Wisconsin, Texas), using page content and link structure as views.
  • Cora: Research papers categorized by subfield, using abstract text and citation links as views.

Results and Discussion

Classification Performance

MVLL-Reg consistently outperformed state-of-the-art global multi-view methods and single-view local methods. On the WebKB "Cornell" dataset, MVLL-Reg achieved an accuracy of 91.25%, significantly higher than the 87.15% achieved by the best combined Laplacian competitor. On the "Texas" dataset, performance reached 94.85%.

Classification results showing MVLL-Reg's superior accuracy across various datasets compared to baselines.

Clustering Performance

In unsupervised tasks, the improvement was even more pronounced. Using Normalized Mutual Information (NMI) as a metric, MVLL-Reg scored 0.5362 on the Cornell dataset, nearly tripling the performance of standard K-means (0.1872) and outperforming other manifold-learning techniques like LLE-Reg.

Clustering results (NMI) highlighting the performance gap between MVLL-Reg and traditional methods.

Key Insights

  • Locality as Capacity Control: By focusing on local regions, the model reduces Rademacher complexity, effectively using unlabeled data to regularize the model through cross-view consistency.
  • Union of Neighbors: Capturing neighbors from all views provides a more robust local structure than any single-view model could achieve.
  • Computational Efficiency: The resulting MVLL-Reg matrix is highly sparse, allowing for efficient scaling to larger datasets.

Conclusion

While the current work focuses on two-view scenarios, the local multi-view philosophy shows great promise. Future work may involve scaling the agreement terms to handle more than two views and exploring the application of this framework to dimensionality reduction and ranking tasks. Despite the need for hyperparameter tuning across multiple views, MVLL-Reg represents a significant step forward in leveraging local data structures for multi-view learning.

Summary updated

Abstract

The idea of local learning, i.e., classifying a particular example based on its neighbors, has been successfully applied to many semi-supervised and clustering problems recently. However, the local learning methods developed so far are all devised for single-view problems. In fact, in many real-world applications, examples are represented by multiple sets of features. In this paper, we extend the idea of local learning to multi-view problem, design a multi-view local model for each example, and propose a Multi-View Local Learning Regularization (MVLL-Reg) matrix. Both its linear and kernel version are given. Experiments are conducted to demonstrate the superiority of the proposed method over several state-of-the-art ones.
BibTeX
@inproceedings{zhang2008multi,
  author = {Dan Zhang and Fei Wang and Changshui Zhang and Tao Li},
  title = {{Multi-View Local Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2008},
  url = {https://openreview.net/forum?id=SJVMKRguZB},
}
Figure