Good Views Are Not Enough: Global Counterexamples for Gaussian-Regularized JEPAs

Anran XuarXiv 2026 · arXiv

TL;DR

This paper provides a rigorous mathematical "reality check" for Joint-Embedding Predictive Architectures (JEPAs). It proves that common Gaussian regularization techniques can actually force models to retain useless noise (nuisance) instead of learning clean, invariant scene representations, and demonstrates that "better" geometric alignment to a scene doesn't necessarily translate to better predictive performance.

What Problem Does This Paper Address?

In self-supervised learning, JEPAs are designed to learn representations by predicting one view of a scene from another. A popular theoretical premise (notably from the "Good Views" paper) suggests that if views share only the "scene" and have independent noise, the model will naturally learn to ignore the noise and focus on the scene.

However, many modern JEPAs add a Gaussian regularization term to ensure the learned representations follow a standard normal distribution. This paper investigates a critical conflict: what happens when the goal of "matching a Gaussian" competes with the goal of "ignoring noise"? The authors show that the model often "cheats" by keeping noise in the representation simply because it helps the output distribution look more Gaussian, even if that noise is useless for the actual prediction task.

Methods

The authors use a combination of mathematical proofs and controlled simulations to provide "globally certified counterexamples." Their approach includes:

  • A Four-State Scalar Model: A simple environment where they can analytically prove that the global optimum must retain noise to satisfy the Gaussian constraint.
  • A 512-Dimensional Model: A high-dimensional setup with exact "whitening" (ensuring features are uncorrelated) to show that the problem persists even in complex, well-conditioned systems.
  • Epps–Pulley (EP) Regularization: They specifically analyze this common statistic used for Gaussian marginal matching.
  • Bayes-Risk Decomposition: A tool used to separate "information loss" (what the model forgot) from "readout difficulty" (how hard it is for a linear head to use what the model remembered).

Datasets

The paper relies on carefully constructed synthetic datasets to allow for "global certification" (proving something is true for all possible encoders in a class):

  • Four-State Task: Observations are indexed by a scene ($S$) and a nuisance ($U$), with uniform probabilities.
  • 512-Dimensional Scene Basis: A model with 513 distinct scenes and independent "cube" nuisances.
  • Binary Token Task: A separate task where a model must predict a label $Y$ based on $(S, U)$, used to compare JEPA representations against Next-Token Prediction (NTP).

Key Insights and Ideas

  • The Marginal-Fit Competition: The model faces a trade-off. Predicting one view from another favors ignoring noise. However, a "scene-only" representation often results in a "clumpy" or atomic distribution. To make this distribution look like a smooth Gaussian, the model "splits" the scene, using resampled noise to fill in the gaps.
  • Scene Splitting: As shown in Figure 1, a "scene-only" encoder produces two sharp peaks. To better match the Gaussian target (the dashed line), the "split encoder" uses nuisance variation to create four peaks, achieving a lower overall objective value despite the "cost" of keeping noise.
Figure 1: Gaussian matching and scene splitting. Panels (a–b) illustrate a scene-only encoder and the four-atom comparator; panel (c) shows how the split encoder achieves a lower objective value than the entire scene-only class.
  • Alignment $\neq$ Usefulness: We often evaluate SSL models by how well they "align" with ground-truth scene variables. The authors show this is a trap: a model can be perfectly aligned geometrically but perform worse on downstream tasks than a model like Next-Token Prediction (NTP), which might have "messier" geometry but retains more task-relevant context.

Key Results

  • Global Failure of Invariance: In the 4-state model, the authors prove that every global minimizer retains nuisance. The "clean" scene-only representation is mathematically excluded from being the optimum.
  • Nuisance under Whitening: Even with exact whitening in 512 dimensions, Gaussian regularization forces the model to retain at least 0.015% nuisance variation.
  • JEPA vs. NTP: In a head-to-head comparison, a whitened JEPA aligned significantly better to the "clean" scene than an NTP model. However, the NTP model was much better at the actual prediction task, incurring 0.2872 nats/token lower error.
  • Information Loss: The risk decomposition reveals that the JEPA's poor performance wasn't just because the "head" couldn't read the data—it was because the information was physically lost through the observation channel during training.

Limitations and Future Work

  • Specific Objectives: The proofs focus on specific "soft" objectives like Epps-Pulley and Gaussian marginal matching. While these are representative of the field, results might vary for other types of contrastive or non-contrastive losses.
  • Synthetic Scope: The counterexamples are mathematically rigorous but use synthetic data. Future work could explore how these "splitting" effects manifest in large-scale models trained on natural images or text.
  • Task Dependence: The authors note that whether retained noise is "nuisance" or "useful context" depends entirely on the downstream task, suggesting that universal "foundation" representations may always face these trade-offs.

Summary updated

Abstract

Good views do not suffice to guarantee scene invariance or uniqueness up to rotation at the global optima of Gaussian-regularized JEPAs. We give globally certified counterexamples showing that the uniqueness premise of the Gaussian extension in Good Views fails for specified soft objectives. Gaussian marginal matching can reward retaining independently resampled observation variation, and this benefit can exceed its prediction cost. In a four-state model, comparison against the entire scene-only class proves that every global minimizer splits a scene and admits an equal-value partner outside its rotation orbit. In a 512-dimensional model with a complete scene basis, a spectral gap, and exact population whitening, pooled Epps-Pulley regularization still forces every global minimizer to retain nuisance and yields positive cross-modal alignment error even after the best rotation when modality nuisances are independent. Better scene alignment can also accompany worse prediction. With matched 512-label readout budgets and 1% observation noise, whitened prediction JEPA aligns better to the scene than next-token prediction but incurs 0.2872 nats/token higher linear-readout test NLL. A Bayes-risk decomposition distinguishes information lost through the observation channel from the fitted readout's difficulty using retained information. These results connect three determinants of learned representations: what views share, what regularization favors, and what a prediction task can use.
BibTeX
@misc{xu2026good,
  author = {Anran Xu},
  title = {{Good Views Are Not Enough: Global Counterexamples for Gaussian-Regularized JEPAs}},
  year = {2026},
  eprint = {2611.12137},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12137},
}
Figure