Beyond Conditional Independence in Multi-View Learning

73 concepts62 papers1 proposal

Imagine you are trying to classify a webpage. You have two distinct sources of information: the text content on the page itself, and the anchor text of the hyperlinks pointing to it from other websites. In machine learning, these separate feature sets are called views, and leveraging them simultaneously is the core of multi-view learning.

To make this problem mathematically tractable, early theoretical frameworks relied on a powerful simplification: the conditional independence assumption. This assumption states that given the true category of the data point (e.g., "Academic Homepage"), the views are entirely independent of one another. While this assumption makes the mathematics elegant, real-world data rarely obeys it. The evolution of multi-view learning is largely a story of moving beyond this assumption to handle the messy, correlated reality of multi-modal data.

The Seductive Elegance of View Independence

Why did researchers assume conditional independence in the first place? The answer lies in how it simplifies semi-supervised learning—learning from a small pool of labeled data and a massive pool of unlabeled data.

In their foundational work, Dasgupta et al. proved that if two views are conditionally independent, their agreement on unlabeled data serves as a reliable proxy for actual classification accuracy. This insight justifies co-training, an algorithmic framework where two separate classifiers are trained on different views. Classifier A (trained on page text) labels unlabeled data for Classifier B (trained on anchor text), and vice versa. If the views are independent, the errors made by Classifier A will look like random noise to Classifier B, allowing the system to bootstrap its way to high accuracy.

The Reality: The Feedback Loop of Death

In practice, the conditional independence assumption is routinely violated. The text on a webpage is deeply correlated with how other people describe it in anchor text; they are shaped by the same underlying semantic concepts.

When views are naturally correlated, the bootstrapping process of co-training breaks down. As observed by Gupta et al. in multi-modal video and caption tracking, correlated views cause the classifiers to make the same mistakes on the same instances. Instead of correcting each other, the models enter an error-reinforcing feedback loop, confirming each other's biases and degrading performance.

This failure mode is particularly acute in complex domains like Natural Language Processing (NLP). For instance, Müller et al. demonstrated that in coreference resolution, core features like gender, number, and semantic class are inherently coupled. Because you cannot cleanly separate these features into independent views, co-training frequently fails to outperform simple single-view supervised baselines.

When datasets lack natural, distinct views, researchers sometimes try to force the issue by artificially splitting a single feature vector into two halves. However, Ng et al. showed that forcing a split on interdependent features leads to extreme hyperparameter sensitivity and highly volatile performance. Furthermore, Wei Wang et al. proved that if the chosen views are too highly correlated or redundant, the theoretical sample-efficiency advantages of multi-view learning completely vanish, causing the system to revert to the performance limitations of a standard single-view learner.

Relaxing the Theory

Recognizing that strict conditional independence is an illusion, theorists have worked to relax the assumption while preserving mathematical guarantees.

Strict Independence  ───►  Expansion Property  ───►  Weak Dependence
(Views never overlap)     (Views mildly overlap)     (Views arbitrarily correlated)

An early breakthrough came from Balcan et al., who replaced strict independence with a weaker topological condition known as the expansion property. Instead of requiring the views to be completely uncorrelated, expansion merely requires that a significant fraction of data points labeled as positive by one view are not completely isolated from the positive points in the other view.

More recently, researchers have pushed these boundaries even further. Diakonikolas et al. designed fast co-training algorithms under "weak dependence" using stream-based active learning, proving that efficient learning is possible even when views exhibit complex, arbitrary correlations.

This theoretical tension persists in modern self-supervised and contrastive learning, where different data augmentations (like cropping or color-jittering an image) are treated as multiple views. As highlighted by Krishnamurthy et al., while existing proofs for contrastive learning still rely heavily on the conditional independence of views, real-world success depends on understanding how these models behave when redundancy between views is only partial.

Algorithmic Solutions: Modeling the Residuals

Rather than trying to find perfectly independent views or ignoring the correlations, modern multi-view learning algorithms explicitly model the shared and unique components of the data.

Shared vs. Private Latent Spaces

One prominent strategy is to untangle the data into what is common and what is unique. Traditional generative models assume that a single shared latent variable explains all views, which ignores view-specific nuances. To fix this, Salzmann et al. introduced Factorized Orthogonal Latent Spaces (FOLS). FOLS mathematically forces the representation into two distinct parts: a shared subspace that captures the correlations between views, and private subspaces that capture the independent, view-specific information.

Input View 1 ──┐
               ├─► [ Shared Latent Space ] (Correlations)
Input View 2 ──┘
               ├─► [ Private Latent Space 1 ] (View 1 Unique Info)
               └─► [ Private Latent Space 2 ] (View 2 Unique Info)

By enforcing orthogonality between these spaces, the model ensures that redundant information isn't counted twice. Similarly, Xue et al. developed frameworks to uncover these shared components, preventing the information loss that occurs when models treat views as entirely independent entities.

Deep Multi-View Networks

When data views interact outside of a simple shared subspace, linear factorization isn't enough. Changqing Zhang et al. developed CPM-Nets (Cross Partial Multi-View Networks), which abandon rigid independence constraints altogether. Instead of assuming views are conditionally independent given a latent variable, they train deep networks to learn a versatile, holistic latent representation that can dynamically reconstruct whatever views are available, directly capturing complex residual correlations.

By shifting the paradigm from assuming independence to modeling correlation, these approaches allow multi-view learning to function reliably in real-world deployments where data streams are inherently entangled.

Key Foundational Work

A Parallel Algorithm for Statistical Belief Refinement and its use in Causal Reasoning (1989) — contributed 1 concept elements to this direction.
On the Relations between Probabilistic Logic and p-CMS (1999) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 59 papers
  2. Version 1 · 2026-10-04 · 56 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_conditional_independence_assumption_view_8562584b,
  title = {conditional independence assumption (view independence assumption)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/conditional-independence-assumption-view-independence-assumption/rdr_8562584b820a4f6b9c4a1e3b518fbf8e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_conditional_independence_assumption_view_8562584b_v2,
  title = {conditional independence assumption (view independence assumption)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/conditional-independence-assumption-view-independence-assumption/rdr_8562584b820a4f6b9c4a1e3b518fbf8e/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake