Cross-Entropy Is All You Need To Invert the Data Generating Process

Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E. Vogt, Randall Balestriero, Wieland Brendel, David Klindt

TL;DR

This paper provides a theoretical foundation for the "linear representation hypothesis," proving that standard supervised learning with cross-entropy loss implicitly inverts the data-generating process. By extending nonlinear Independent Component Analysis (ICA) theory, the authors show that classification models recover ground-truth latent factors of variation up to a linear transformation, explaining why interpretable features can often be linearly decoded from neural networks.

What Problem Does This Paper Address?

Deep learning is "unreasonably effective," yet we still lack a comprehensive theory explaining why. We often observe that neural networks learn representations where related concepts are linearly separated or where simple vector arithmetic (like the famous $king - man + woman \approx queen$) works. This is known as the linear representation hypothesis.

While recent advances in self-supervised learning (SSL) and nonlinear ICA have shown that certain models can recover the underlying "latent" structure of data, these results haven't fully translated to the most common setting in machine learning: standard supervised classification using cross-entropy loss. The authors aim to bridge this gap, providing a theoretical explanation for why supervised models learn such structured, interpretable, and transferable representations.

Methods

The authors tackle this by building a "genealogy" of classification methods, connecting general theoretical frameworks to practical supervised learning.

  1. Cluster-Centric Data Generating Process (DGP): They propose a new model for how data is created. In this model, semantic classes are represented by "cluster vectors" on a hypersphere. Individual data points are generated by sampling latents from a von Mises-Fisher (vMF) distribution—essentially a Gaussian distribution restricted to the surface of a sphere—centered around these cluster vectors.
  2. Theoretical Extension of DIET: They first analyze DIET (Discrete Instance Embedding Training), a parametric instance discrimination method. They prove that DIET identifies both the latent variables and the cluster vectors up to a linear transformation.
  3. Supervised Learning as a Special Case: The core breakthrough is proving that standard supervised classification is simply a version of DIET where the "instance labels" are replaced by "semantic class labels." Under their proposed DGP, they prove that the learned representation $f(x)$ is a linear map of the true latent $z$.
Figure 1: DIET (Ibrahim et al., 2024) learns identifiable features: given N samples and a d−dimensional latent representation, DIET learns a linear (N × d)−dimensional classification head W on top of a nonlinear encoder f through an instance discrimination objective (1). For unit-normalized f(xn), DIET maps samples and their augmentations close to the cluster vector vc corresponding to the class—as if sampled from a von Mises-Fisher (vMF) distribution, centered around vc. For duplicate samples, i.e., matching class labels, the corresponding rows of W will be the same, as shown for x1 and xi with w1 = wi.

Datasets

The authors validate their theory across three levels of complexity:

  • Numerical Simulations: Synthetic data generated strictly according to their vMF cluster-centric assumptions to verify the mathematical proofs.
  • DisLib Benchmark: A standard disentanglement suite including dSprites, Shapes 3D, MPI 3D, Cars 3D, and smallNORB. These datasets have known ground-truth factors (like object scale, position, or elevation).
  • ImageNet-X: A real-world dataset based on ImageNet that includes human-annotated binary "proxy" factors of variation (e.g., whether an object is blocked, its color, or its pose).

Key Insights and Ideas

  • Cross-Entropy as an Inverter: The most significant "aha moment" is that the cross-entropy objective, by forcing a model to distinguish between classes, effectively forces the model to "invert" the process that generated the data.
  • The Genealogy of Classification: The paper provides a beautiful map (Figure 2) showing how complex, general theories like Generalized Contrastive Learning (GCL) simplify down into ubiquitous methods like InfoNCE and standard Supervised Learning.
Figure 2: The simplified genealogy of cross-entropy-based classification methods (cf. Tab. 1 for details): The labeled arrows express how to go from general to special methods. (a) The most general auxiliary-variable ICA framework, Generalized Contrastive Learning (GCL) (Hyvarinen et al., 2019), yields Time-Contrastive Learning (TCL) (Hyvarinen & Morioka, 2016) as the special case when the latent conditional is assumed to come from an exponential family (of order one) with a scalar auxiliary variable; (b) TCL relates to non-unit-normalized DIET by further restricting the latent conditional to a vMF distribution; (c) if the neural network used in InfoNCE is partitioned into a linear classifier head and a backbone, the marginal is assumed to be a vMF instead of uniform, we get the unit-normalized version of DIET; (d) if the labeling function in DIET is assumed to assign the semantic class labels to the samples, we get classic supervised training
  • Linear Identifiability: Unlike previous ICA results that often only guaranteed recovery up to "element-wise nonlinearities," this work proves linear identifiability. This means the learned features are a simple linear transformation of the true features, which is why linear readouts work so well in transfer learning.

Key Results

  • Synthetic Verification: In settings matching their theoretical assumptions, the models achieved $R^2$ scores near 100% for recovering ground-truth latents.
  • DisLib Performance: On benchmarks like dSprites and Shapes 3D, simple classification tasks recovered latent structures (scale, position) with Pearson correlations often exceeding 0.95, significantly outperforming raw input baselines.
  • Real-World Evidence: Using pretrained ResNet50 and ViT models on ImageNet, the authors showed that human-annotated factors in ImageNet-X could be linearly decoded with high accuracy, far above chance levels. For example, factors like "darker," "brighter," and "pose" were highly decodable from standard supervised representations.

Limitations and Future Work

  • DGP Assumptions: The theory relies on the data being somewhat "clustered" and the latents living on a hypersphere. While this covers many cases, it might not perfectly model data with complex non-Euclidean topologies (like 3D rotations).
  • Class Balance: The current proof for supervised learning assumes a uniform distribution of classes, which doesn't account for the long-tail distributions found in many real-world datasets.
  • Proxy Factors: For real-world data, we don't know the "true" latents. ImageNet-X uses human proxies, which are helpful but might not capture the full underlying data-generating process.
  • Future Directions: The authors suggest extending these insights to other forms of supervised learning and testing the scalability of these theoretical results on even larger, more complex models and datasets.

Summary updated

Abstract

Supervised learning has become a cornerstone of modern machine learning, yet a comprehensive theory explaining its effectiveness remains elusive. Empirical phenomena, such as neural analogy-making and the linear representation hypothesis, suggest that supervised models can learn interpretable factors of variation in a linear fashion. Recent advances in self-supervised learning, particularly nonlinear Independent Component Analysis, have shown that these methods can recover latent structures by inverting the data generating process. We extend these identifiability results to parametric instance discrimination, then show how insights transfer to the ubiquitous setting of supervised learning with cross-entropy minimization. We prove that even in standard classification tasks, models learn representations of ground-truth factors of variation up to a linear transformation. We corroborate our theoretical contribution with a series of empirical studies. First, using simulated data matching our theoretical assumptions, we demonstrate successful disentanglement of latent factors. Second, we show that on DisLib, a widely-used disentanglement benchmark, simple classification tasks recover latent structures up to linear transformations. Finally, we reveal that models trained on ImageNet encode representations that permit linear decoding of proxy factors of variation. Together, our theoretical findings and experiments offer a compelling explanation for recent observations of linear representations, such as superposition in neural networks. This work takes a significant step toward a cohesive theory that accounts for the unreasonable effectiveness of supervised deep learning.
BibTeX
@inproceedings{reizinger2025cross,
  author = {Patrik Reizinger and Alice Bizeul and Attila Juhos and Julia E. Vogt and Randall Balestriero and Wieland Brendel and David Klindt},
  title = {{Cross-Entropy Is All You Need To Invert the Data Generating Process}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=hrqNOxpItr},
}
Figure