TL;DR
Standard multimodal alignment objectives (like those used in CLIP) are fundamentally flawed when modalities are informationally asymmetric, as they force the model to discard unique, task-relevant information. This paper introduces Decomposed JEPA (D-JEPA), an architecture that factorizes representations into an aligned "shared core" and modality-specific "private fibers," successfully recovering the synergistic information necessary for complex compositional reasoning.
What Problem Does This Paper Address?
In multimodal learning, the "redundancy premise" is the dominant assumption: that the most important information is what is shared between modalities (e.g., an image and its exhaustive caption). Consequently, researchers use cross-modal alignment to force representations from different modalities to be as similar as possible.
However, the author argues that in many real-world tasks, modalities are informationally asymmetric. Consider Visual Question Answering (VQA): an image contains a dense field of visual data, while a question is a selective relational operator (e.g., "How many blue cubes are behind the cylinder?"). The question doesn't describe the scene; it interrogates it.
When you apply standard alignment to these asymmetric pairs, the model treats any information not present in both as noise. This leads to "semantic collapse," where the model discards the fine-grained visual details and specific linguistic referents needed to actually answer the question. The paper formalizes this as the Alignment Redundancy Bound, proving that alignment-only objectives can only ever capture the redundant "intersection" of information, annihilating unique and synergistic data.
Methods
To escape this bound, the authors propose Decomposed JEPA (D-JEPA). Instead of projecting everything into a single invariant vector, D-JEPA factorizes the latent space of each modality into two orthogonal subspaces:
- Shared Core ($c$): Captures the redundant information that both modalities share.
- Private Fibers ($u$): Captures modality-specific information that alignment would otherwise discard.
The architecture is trained using a multi-part objective:
- Scale-Invariant Common Alignment: Uses unit-normalized cosine Dirichlet energy to align the shared cores without triggering magnitude collapse.
- Contextual JEPA Fiber Prediction: Uses a Joint-Embedding Predictive Architecture (JEPA) approach. Lightweight predictors are trained to reconstruct target-view representations from the context latents, ensuring the private fibers ($u$) preserve task-relevant structure.
- Asymmetric Cross-Subspace Independence: A penalty that prevents private information from leaking into the shared code, while crucially allowing the private fibers of different modalities to coordinate for compositional reasoning.
Datasets
The authors evaluate D-JEPA on two main benchmarks:
- Synthetic Controlled Ground-Truth Benchmark: A custom dataset where the information-theoretic "atoms" (Shared, Private, and Synergistic) are mathematically known. This allows for precise auditing of what the model actually learns.
- CLEVR: A popular diagnostic VQA dataset consisting of 3D-rendered scenes and complex relational questions. It is a prime example of informational asymmetry, as questions require specific attributes (color, shape) and relational logic (counting, spatial reasoning) that are not "shared" in a simple alignment sense.
Key Insights and Ideas
- The Death of Synergy: The paper proves that alignment-only training doesn't just lose unique info; it destroys synergy—information that can only be derived by looking at both modalities together. In alignment-only models, the "synergy gap" often becomes negative, meaning the model performs worse with two modalities than with one.
- Factorization as an Escape Hatch: By explicitly providing a "private" subspace, D-JEPA allows the model to satisfy the alignment objective in the shared core while safely stashing the "un-alignable" but necessary details in the fibers.
- Causal Localization: Through "fiber-swap" interventions (replacing one modality's private fiber with a random donor's), the authors prove that the information needed for reasoning is causally localized in these private fibers. Swapping the private text fiber $u_2$ flipped 71.36% of predictions on CLEVR.
- Latent Disentanglement: t-SNE visualizations show that D-JEPA successfully organizes information into well-separated geometric manifolds, whereas alignment-only models collapse everything onto a single shared manifold.
Key Results
- CLEVR Performance: D-JEPA achieved 39.61% accuracy, a massive +18.36 percentage point gain over the alignment-only baseline (21.25%). It even outperformed a baseline augmented with reconstruction (+11.65 pp gain).
- Restoring Synergy: On the synthetic benchmark, alignment-only models collapsed to near-chance (48.33%) on synergistic factors. D-JEPA recovered these factors almost entirely (84.20%), restoring a positive synergy gap.
- Information Recovery: Information-theoretic audits (Probe-PID) showed that while alignment-only training obliterated mutual information (falling from 2.293 bits to 0.242 bits), D-JEPA restored it to 1.938 bits (84.5% of the theoretical ceiling).
Limitations and Future Work
- Frozen Backbones: The current study uses frozen encoders (DINOv2 and BERT). The authors note that end-to-end finetuning or training from scratch with D-JEPA objectives remains an open area.
- The Performance Gap: While D-JEPA significantly beats alignment baselines, it hasn't yet closed the gap to the "raw" frozen backbone ceiling (45.57% on CLEVR).
- Scaling: The evaluations were performed at a reduced scale. Testing D-JEPA on massive web-scale datasets (like LAION) or in continuous robotic action spaces is a suggested next step.
- Capacity Allocation: The authors suggest that future work could develop adaptive mechanisms to dynamically adjust how much latent capacity is allocated to shared vs. private subspaces based on the "information density" of the input.
Summary updated