Asymmetric Multimodal Learning with Decomposed JEPAs

Yusuf ShihataarXiv 2026 · arXiv

TL;DR

Standard multimodal alignment objectives (like those used in CLIP) are fundamentally flawed when modalities are informationally asymmetric, as they force the model to discard unique, task-relevant information. This paper introduces Decomposed JEPA (D-JEPA), an architecture that factorizes representations into an aligned "shared core" and modality-specific "private fibers," successfully recovering the synergistic information necessary for complex compositional reasoning.

What Problem Does This Paper Address?

In multimodal learning, the "redundancy premise" is the dominant assumption: that the most important information is what is shared between modalities (e.g., an image and its exhaustive caption). Consequently, researchers use cross-modal alignment to force representations from different modalities to be as similar as possible.

However, the author argues that in many real-world tasks, modalities are informationally asymmetric. Consider Visual Question Answering (VQA): an image contains a dense field of visual data, while a question is a selective relational operator (e.g., "How many blue cubes are behind the cylinder?"). The question doesn't describe the scene; it interrogates it.

When you apply standard alignment to these asymmetric pairs, the model treats any information not present in both as noise. This leads to "semantic collapse," where the model discards the fine-grained visual details and specific linguistic referents needed to actually answer the question. The paper formalizes this as the Alignment Redundancy Bound, proving that alignment-only objectives can only ever capture the redundant "intersection" of information, annihilating unique and synergistic data.

Methods

To escape this bound, the authors propose Decomposed JEPA (D-JEPA). Instead of projecting everything into a single invariant vector, D-JEPA factorizes the latent space of each modality into two orthogonal subspaces:

  1. Shared Core ($c$): Captures the redundant information that both modalities share.
  2. Private Fibers ($u$): Captures modality-specific information that alignment would otherwise discard.
Figure 1: Alignment under informational asymmetry. (a) When two modalities are not redundant views of one state (e.g., scene context vs. relational query), cross-modal invariance projects both onto their redundant intersection Red, destroying unique referents Unqm and synergistic bindings Syn. (b) D-JEPA factorizes the latent space into an aligned shared core c and modality-specific private fibers um, anchored by contextual masked prediction.

The architecture is trained using a multi-part objective:

  • Scale-Invariant Common Alignment: Uses unit-normalized cosine Dirichlet energy to align the shared cores without triggering magnitude collapse.
  • Contextual JEPA Fiber Prediction: Uses a Joint-Embedding Predictive Architecture (JEPA) approach. Lightweight predictors are trained to reconstruct target-view representations from the context latents, ensuring the private fibers ($u$) preserve task-relevant structure.
  • Asymmetric Cross-Subspace Independence: A penalty that prevents private information from leaking into the shared code, while crucially allowing the private fibers of different modalities to coordinate for compositional reasoning.

Datasets

The authors evaluate D-JEPA on two main benchmarks:

  • Synthetic Controlled Ground-Truth Benchmark: A custom dataset where the information-theoretic "atoms" (Shared, Private, and Synergistic) are mathematically known. This allows for precise auditing of what the model actually learns.
  • CLEVR: A popular diagnostic VQA dataset consisting of 3D-rendered scenes and complex relational questions. It is a prime example of informational asymmetry, as questions require specific attributes (color, shape) and relational logic (counting, spatial reasoning) that are not "shared" in a simple alignment sense.

Key Insights and Ideas

  • The Death of Synergy: The paper proves that alignment-only training doesn't just lose unique info; it destroys synergy—information that can only be derived by looking at both modalities together. In alignment-only models, the "synergy gap" often becomes negative, meaning the model performs worse with two modalities than with one.
  • Factorization as an Escape Hatch: By explicitly providing a "private" subspace, D-JEPA allows the model to satisfy the alignment objective in the shared core while safely stashing the "un-alignable" but necessary details in the fibers.
  • Causal Localization: Through "fiber-swap" interventions (replacing one modality's private fiber with a random donor's), the authors prove that the information needed for reasoning is causally localized in these private fibers. Swapping the private text fiber $u_2$ flipped 71.36% of predictions on CLEVR.
  • Latent Disentanglement: t-SNE visualizations show that D-JEPA successfully organizes information into well-separated geometric manifolds, whereas alignment-only models collapse everything onto a single shared manifold.
Figure 2: t-SNE latent geometry of D-JEPA subspaces. Held-out latents of the four learned codes: shared c1 (vision, blue) and c2 (text, teal) exhibit close cross-modal alignment, while private fibers u1 (vision, orange) and u2 (text, purple) occupy isolated, non-overlapping subspaces.

Key Results

  • CLEVR Performance: D-JEPA achieved 39.61% accuracy, a massive +18.36 percentage point gain over the alignment-only baseline (21.25%). It even outperformed a baseline augmented with reconstruction (+11.65 pp gain).
  • Restoring Synergy: On the synthetic benchmark, alignment-only models collapsed to near-chance (48.33%) on synergistic factors. D-JEPA recovered these factors almost entirely (84.20%), restoring a positive synergy gap.
  • Information Recovery: Information-theoretic audits (Probe-PID) showed that while alignment-only training obliterated mutual information (falling from 2.293 bits to 0.242 bits), D-JEPA restored it to 1.938 bits (84.5% of the theoretical ceiling).

Limitations and Future Work

  • Frozen Backbones: The current study uses frozen encoders (DINOv2 and BERT). The authors note that end-to-end finetuning or training from scratch with D-JEPA objectives remains an open area.
  • The Performance Gap: While D-JEPA significantly beats alignment baselines, it hasn't yet closed the gap to the "raw" frozen backbone ceiling (45.57% on CLEVR).
  • Scaling: The evaluations were performed at a reduced scale. Testing D-JEPA on massive web-scale datasets (like LAION) or in continuous robotic action spaces is a suggested next step.
  • Capacity Allocation: The authors suggest that future work could develop adaptive mechanisms to dynamically adjust how much latent capacity is allocated to shared vs. private subspaces based on the "information density" of the input.

Summary updated

Abstract

Cross-modal alignment is the standard objective for building multimodal representations. We show that alignment as a \emph{sole} objective is lossy wherever modalities are informationally asymmetric---where neither can express what the other carries. Visual question answering is an obvious case: a question is a selective relational operator probing a dense visual field whose fine-grained features no query recovers, so what alignment preserves is not what the task requires. We prove an \textbf{Alignment Redundancy Bound}: alignment-only Dirichlet energy minimization retains at most the redundant information atom $\Red(Z; X_1, X_2)$, annihilating private modality-specific structure $\Unq(Z; X_m)$ and cross-modal synergy $\Syn(Z; X_1, X_2)$. We introduce \textbf{Decomposed JEPA (D-JEPA)}, which factorizes latents into an aligned shared core and private modality-specific fibers. The shared core retains most of the structure alignment targets; the private fibers recover what alignment discards, via contextual masked prediction and asymmetric cross-subspace independence. Under information asymmetry, alignment-only training undergoes catastrophic semantic collapse on compositional reasoning benchmarks, surviving only on binary queries. D-JEPA averts this collapse, substantially improving downstream accuracy over both alignment-only and alignment-with-reconstruction baselines, and restoring a positive multimodal synergy gap in place of the baseline's inverted gap. Information-theoretic audits confirm that D-JEPA recovers most of the mutual information available from a frozen backbone, and causal fiber-swap interventions demonstrate that the recovered information is causally localized in the private fibers.
BibTeX
@misc{shihata2026asymmetric,
  author = {Yusuf Shihata},
  title = {{Asymmetric Multimodal Learning with Decomposed JEPAs}},
  year = {2026},
  eprint = {2611.16639},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16639},
}
Figure