A Unified Learning Scheme: Bayesian-Kullback Ying-Yang Machine

Lei Xu1995

TL;DR

This paper introduces the "Ying-Yang Machine," a novel unified learning framework that bridges supervised and unsupervised learning by leveraging two complementary Bayesian representations of joint data density. This unified approach not only reconciles existing machine learning algorithms but also proposes new models and offers a deeper understanding of their relationships.

What Problem Does This Paper Address?

The field of machine learning has seen a proliferation of diverse learning algorithms, each with its own strengths and applications. This diversity, while powerful, can lead to fragmentation and a lack of clear understanding of how different methods relate to each other. The authors identify a need for a unified theoretical framework that can encompass and explain existing learning paradigms, facilitate cross-pollination of ideas, and guide the development of new models. Specifically, they aim to unify supervised and unsupervised learning methods, including maximum likelihood, information preservation, EM algorithms, information geometry, and Helmholtz machines, under a single, coherent scheme.

Methods

The core of the proposed "Bayesian-Kullback Ying-Yang Machine" lies in its use of two complementary Bayesian representations for the joint probability distribution $P(x, y)$ of input data $x$ and representation $y$. These are:

  • YANG (male) part (Model $M_1$): Learns to encode or recognize, mapping input $x$ to a representation $y$. This is represented as $P_{M_1}(x, y) = P_{M_1}(y|x)P_{M_1}(x)$.
  • YING (female) part (Model $M_2$): Learns to decode or generate, mapping representation $y$ back to input $x$. This is represented as $P_{M_2}(x, y) = P_{M_2}(x|y)P_{M_2}(y)$.

The learning objective is framed around minimizing or maximizing the Kullback divergence between these two representations. The framework is highly flexible, allowing for various choices in how these components are parameterized or fixed, leading to different learning models. The paper also introduces an "Alternative Minimization" (ALTMIN) procedure for optimizing these models, which is analogous to the EM algorithm.

Datasets

Not discussed in the paper. The paper focuses on the theoretical framework and its unification of existing learning models, rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • Unified Framework: The Ying-Yang Machine provides a powerful unifying principle for a wide array of learning algorithms, from classical maximum likelihood to more recent Helmholtz machines. This unification helps demystify the relationships between these methods.
  • Complementary Representations: The core idea of using two complementary Bayesian representations (encoding/recognition and decoding/generation) that form a cycle ($x \to y \to x$) is elegant and provides a natural way to model joint distributions.
  • "Marital Status" of Models: The paper frames the interaction between the two representations as "marital status" (marry, divorce, chase), which corresponds to different optimization objectives (minimization/maximization of Kullback divergence). This provides a novel way to categorize and understand learning algorithms.
  • New Perspectives on Existing Models: The framework offers new interpretations for established models like the Helmholtz machine, clarifying potential confusions and highlighting their connections to other learning paradigms.
  • Guidance for New Models: By systematically exploring the combinations of components and optimization objectives, the Ying-Yang Machine naturally suggests new learning models and variants, as demonstrated in the "New Learning Models" section.

Key Results

The paper's primary contribution is the theoretical unification of existing learning models and the proposal of new ones. Specific results include:

  • Demonstration that the Ying-Yang Machine, under certain conditions, reduces to:
  • The EM algorithm for mixture models.
  • Maximum information preservation (Informax) learning.
  • Helmholtz machine.
  • Maximum likelihood and least squares learning.
  • MDL auto-encoders and Multi-sets modeling.
  • Proposal of new learning models, particularly for supervised learning, that suggest learning by minimum entropy or a mix of maximum likelihood and minimum entropy, offering alternatives to pure maximum likelihood.
  • Identification of three VQ variants and criteria for selecting the number of clusters when $P_{M_2}(x|y)$ is Gaussian.
  • A robust PCA method with a criterion for determining subspace dimension.

Limitations and Future Work

  • Experimental Validation: While the paper proposes new models and theoretical connections, it acknowledges that experimental validation is ongoing. The paper states, "We are currently conducting experiments on comparison several of the above new models against their existing counterparts."
  • Other "Marital Statuses": The paper focuses primarily on the "marry" and "divorce" statuses of the Ying-Yang pairs, suggesting that further study is needed for other statuses like "chasing" and "escaping."
  • Parameter Selection: For some proposed new models, the paper suggests heuristic approaches for parameter selection (e.g., for H-m-W status), but notes that their effectiveness needs to be experimentally tested.
  • Temporal Patterns: The paper briefly mentions extensions to temporal patterns but does not delve into them in detail, indicating a potential area for future research.

Summary updated

Abstract

A Bayesian-Kullback learning scheme, called Ying-Yang Machine, is proposed based on the two complement but equivalent Bayesian representations for joint density and their Kullback divergence. Not only the scheme unifies existing major supervised and unsupervised learnings, including the classical maximum likelihood or least square learning, the maximum information preservation, the EM & em algorithm and information geometry, the recent popular Helmholtz machine, as well as other learning methods with new variants and new results; but also the scheme provides a number of new learning models.
BibTeX
@misc{xu1995unified,
  author = {Lei Xu},
  title = {{A Unified Learning Scheme: Bayesian-Kullback Ying-Yang Machine}},
  year = {1995},
}
Figure