Much of our theoretical understanding of deep learning is built on a "mathematical convenience": the Mean Squared Error (MSE) loss. Because MSE is quadratic, it yields linear gradients and allows for closed-form solutions that make complex phenomena like training dynamics and feature geometry tractable. However, modern classification—the backbone of Large Language Models and computer vision—relies almost exclusively on Cross-Entropy (CE) loss.
Researchers are currently working to bridge this "loss function gap," investigating whether the elegant proofs derived for squared loss actually hold in the non-quadratic, often divergent landscapes of classification objectives.
The Tractability Trap
In theoretical physics, researchers often joke about the "spherical cow"—a simplified model that makes calculations possible but ignores reality. In deep learning theory, MSE is that spherical cow. As Han et al. demonstrate, the mathematical tractability of MSE allows us to derive exact dynamics for how features evolve during training.
This convenience extends to almost every corner of the field:
- Optimization Landscapes: Kawaguchi and Zhou provide foundational proofs that deep linear networks lack poor local minima, but these guarantees are strictly tied to the properties of squared loss.
- Kernel Methods: The Neural Tangent Kernel (NTK) framework, which explains how infinite-width networks behave, is most naturally expressed through squared loss. Shwartz-Ziv et al. and Franceschi et al. note that extending these insights to the logistic or cross-entropy losses used in practice requires entirely different analytical machinery.
The Case of Neural Collapse
The tension between these two loss functions is most visible in the study of Neural Collapse (NC). This phenomenon occurs at the end of training when the features of all samples in a class "collapse" into a single point, and these points arrange themselves into a perfectly symmetric geometric structure (a simplex).
While NC was first observed empirically with Cross-Entropy, the first rigorous proofs of its emergence, such as those by Han et al., were restricted to MSE. The challenge is that CE behaves fundamentally differently: while MSE has a well-defined global minimum at zero, CE loss on separable data is minimized only as the weights grow to infinity. Tirer et al. have begun bridging this gap by using perturbation analysis to show how layer-by-layer variability reduction occurs even under the messier reality of CE.
Why the Loss Function Matters
The choice of loss function isn't just a detail; it changes the "inductive bias" of the model—the hidden rules that determine which solution a network picks among many possibilities.
- Convergence Speed: Tachet des Combes et al. highlight that different losses influence the growth of model confidence and the speed of convergence. Some losses may suffer from "gradient starvation," where the network ignores useful features because a few dominant ones have already satisfied the objective.
- Regularization Behavior: Even standard techniques like Dropout behave differently depending on the loss. Helmbold et al. prove that the specific regularization penalties imposed by stochastic training are sensitive to whether the objective is quadratic or probabilistic.
- Geometry of Minima: In classification, the global minimum for separable data exists at infinity. This makes the study of "critical points" (where the gradient is zero) significantly harder. Nguyen et al. argue that while width smooths the optimization landscape for MSE, the non-separable nature of CE requires new mathematical tools to track how models reach these distant optima.
Toward a Unified Theory
Recent work suggests that we are moving toward a more general framework that encompasses both worlds. Mulgund et al. have shown that by viewing loss functions through the lens of Bregman divergences, we can extend theories of "weak-to-strong generalization" beyond simple regression to complex classification tasks. Similarly, Wang et al. have extended meta-learning convergence proofs to broader regimes, though they note that a full alignment with practical Cross-Entropy usage remains an open frontier.
The goal of this research direction is to ensure that our theoretical "maps" of deep learning actually describe the terrain where modern models live, rather than just the simplified landscapes where the math is easiest.