Multi-View Learning over Structured and Non-Identical Outputs

Kuzman Ganchev, João Graça, John Blitzer, Ben Taskar

TL;DR

This paper introduces a novel probabilistic framework for multi-view learning that uses "stochastic agreement" between different data views as a regularization technique. This approach allows models trained on different views to learn from each other, leading to improved performance, especially when labeled data is scarce. The method is flexible, handling both structured and non-identical output spaces, and demonstrates strong results on various natural language processing tasks.

What Problem Does This Paper Address?

In many real-world machine learning scenarios, we have abundant unlabeled data but only a limited amount of labeled data. Multi-view learning aims to leverage this situation by assuming that different "views" of the same data (e.g., different feature sets, different modalities) can each provide a good prediction of the target label. The challenge is to effectively combine the information from these views to build a more robust and accurate model than what could be achieved from any single view alone. Existing methods often struggle with cases where the views have different output structures or label sets, or when the agreement between views is not perfect. This paper tackles the problem of how to effectively regularize models from multiple views to encourage agreement, thereby improving generalization, particularly in scenarios with limited labeled data and potentially complex data structures.

Methods

The core contribution of this paper is a probabilistic agreement framework that minimizes the Bhattacharyya distance between the predictions of models trained on different views. This distance metric is a symmetric measure of similarity between probability distributions. The authors propose to optimize this objective using a constrained Expectation-Maximization (EM) algorithm.

Key methodological contributions include:

  • Stochastic Agreement Regularization: The paper introduces a novel regularizer based on minimizing the Bhattacharyya distance between the output distributions of different views. This encourages the models to agree on their predictions for unlabeled data.
  • Probabilistic Framework: The approach is grounded in probabilistic modeling, allowing for principled integration of different views.
  • Constrained EM Optimization: The authors adapt the EM algorithm to optimize the proposed objective, which is convex in each model's parameters and the agreement distribution separately, but not jointly.
  • Handling Structured and Non-Identical Outputs: A significant innovation is the extension of the framework to handle structured prediction tasks (like sequence labeling) and scenarios where the views have different label sets or output structures. This is achieved by reformulating the agreement constraints.
  • Closed-form Solutions: For certain cases, particularly with undirected graphical models and unstructured data, the paper derives closed-form solutions for computing the agreement distribution, simplifying optimization.

Datasets

The paper evaluates its method on several benchmark datasets:

  • Sentiment Classification: Data from Blitzer et al. (2007) is used for transfer learning experiments, where views are created by randomly splitting features.
  • Named Entity Disambiguation: The 2003 CONLL shared task data is used, with views based on content and context features.
  • Partial Agreement Experiments: The 20-newsgroups corpus is used to simulate scenarios with partial agreement, specifically focusing on newsgroups related to computer hardware and politics/baseball.
  • Noun Phrase Chunking: The English training portion of the CONLL 2000 shared task is used for structured prediction experiments.
  • Named Entity Resolution: Experiments are conducted on the 2003 CONLL shared task data for structured partial agreement.

Key Insights and Ideas

  • Agreement as Regularization: The fundamental insight is that forcing different views of the data to agree on unlabeled instances acts as a powerful regularizer, preventing overfitting and improving generalization, especially when labeled data is scarce.
  • Bhattacharyya Distance is Effective: The choice of Bhattacharyya distance as the agreement metric proves effective for optimizing log-loss objectives and leads to better performance than other co-regularization approaches.
  • Flexibility for Complex Scenarios: The ability to handle non-identical output spaces and partial agreement is a major advantage, opening up multi-view learning to a wider range of real-world problems where views might not perfectly align.
  • Joint Learning vs. Opinion Pools: Experiments suggest that the benefit of their method comes from the joint objective function (co-regularization) rather than solely from using the agreement mechanism as an opinion pool for inference.

Key Results

  • Improved Performance: The proposed "SAR" (Stochastic Agreement Regularization) method consistently outperforms or matches state-of-the-art methods like CoBoosting and two-view Perceptron on sentiment classification, named entity disambiguation, and structured prediction tasks.
  • Transfer Learning Success: In transfer learning for sentiment classification, SAR achieved the best performance on 6 out of 12 tasks, outperforming SCL (4 tasks) and CoBoosting (1 task).
  • Structured Prediction Gains: For noun phrase chunking, SAR showed significant gains over monolithic CRFs and two-view Perceptron, especially with limited training data.
  • Partial Agreement Benefits: In partial agreement scenarios (e.g., different label sets), the method demonstrated improvements in reducing confusion between similar categories.

Limitations and Future Work

  • Computational Complexity: While closed-form solutions exist for some cases, optimizing for structured problems with non-identical outputs can still be computationally intensive, requiring iterative gradient steps.
  • Generalization to More Than Two Views: The paper focuses on two views but notes that the framework can be generalized to multiple views. Future work could explore optimal ways to weigh these multiple views, similar to logarithmic opinion pools.
  • Application to Complex Structures: Extending the framework to problems where proposal distributions are difficult to compute in closed form (e.g., combining dependency and phrase structure parsers) is a promising direction.
  • Theoretical Understanding: Investigating the conditions under which multi-view co-regularization frameworks like theirs lead to asymptotically faster convergence than monolithic models is an open research question.

Summary updated

Abstract

In many machine learning problems, labeled training data is limited but unlabeled data is ample. Some of these problems have instances that can be factored into multiple views, each of which is nearly sufficent in determining the correct labels. In this paper we present a new algorithm for probabilistic multi-view learning which uses the idea of stochastic agreement between views as regularization. Our algorithm works on structured and unstructured problems and easily generalizes to partial agreement scenarios. For the full agreement case, our algorithm minimizes the Bhattacharyya distance between the models of each view, and performs better than CoBoosting and two-view Perceptron on several flat and structured classification problems.
BibTeX
@inproceedings{ganchev2008multi,
  author = {Kuzman Ganchev and João Graça and John Blitzer and Ben Taskar},
  title = {{Multi-View Learning over Structured and Non-Identical Outputs}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2008},
}
Figure