Abstract
This paper investigates the optimization landscape of deep neural networks trained with Mean Squared Error (MSE) loss during the terminal phase of training. We prove that under the unconstrained feature model, the MSE loss landscape is "benign," characterized by the absence of spurious local minima and the presence of only global minimizers and strict saddle points. These global minimizers correspond to "Neural Collapse" (NC) solutions, where features of the same class collapse to a single point and class means form a Simplex Equiangular Tight Frame (ETF). Furthermore, we provide a geometric explanation for the performance gap between vanilla and rescaled MSE, demonstrating how rescaling aligns gradients to improve convergence and generalization.
Introduction
In modern deep learning, the "terminal phase of training" (TPT)—where training error approaches zero—reveals a phenomenon known as Neural Collapse (NC). NC is characterized by four properties: within-class feature collapse, convergence of class means to a Simplex ETF, alignment of classifiers with class means, and simplification to a nearest-class-center decision rule. While NC is well-documented for Cross-Entropy (CE) loss, its emergence under MSE loss has lacked rigorous theoretical grounding. This study addresses whether MSE consistently leads to NC solutions and if its optimization landscape is free of problematic local minima.
Theoretical Framework: The Unconstrained Feature Model
To analyze the landscape, we employ the Unconstrained Feature Model (UFM). In this framework, the last-layer features are treated as independent optimization variables. This simplification is justified by the extreme over-parameterization of modern networks, which allows them to map inputs to arbitrary points in the feature space. Our analysis focuses on the global landscape, proving that every critical point is either a global minimum (representing a perfect NC state) or a "strict saddle point" with negative curvature, which standard optimizers like SGD can easily escape.
Geometric Insights and the Phase Transition
A key finding is the phase transition related to the feature dimension $d$. If $d \ge K-1$ (where $K$ is the number of classes), the network achieves a perfect Simplex ETF. If $d < K-1$, the features collapse to the best rank-$d$ approximation of this symmetric structure.
The paper also explains why "vanilla" MSE often underperforms compared to CE on large-scale datasets. By analyzing the loss in polar coordinates (angle $\theta$ and length $s$), we observe that vanilla MSE is "steep" regarding feature length but "flat" regarding the angle. Since classification accuracy depends primarily on the angle between features and class weights, this flatness leads to slow convergence.
The Power of Rescaled MSE
To overcome the limitations of vanilla MSE, the authors explore a "rescaling trick." By weighting the loss of the "correct" class more heavily, the gradient is realigned to be perpendicular to the decision boundary. This modification makes the MSE landscape behave more like the CE landscape, significantly improving performance on datasets like miniImageNet. This explains why rescaled MSE can match or even beat industry-standard losses.
Experimental Validation
We validated our theoretical claims using ResNet18 on CIFAR10 and miniImageNet:
- Algorithm Independence: Convergence to NC solutions was observed across various optimizers (SGD, Adam, L-BFGS), confirming the benign nature of the landscape.
- Efficiency: We demonstrated that fixing the last-layer classifier as a static Simplex ETF does not degrade accuracy, potentially reducing trainable parameters.
- Rescaling Efficacy: On miniImageNet, a clear correlation was found between the rescaling parameters, the degree of Neural Collapse, and improved test accuracy.
Conclusion and Limitations
This work provides the first global landscape analysis for MSE loss in the context of Neural Collapse, proving its optimality and benign nature. While the UFM provides deep insights, the model remains an approximation of full, multi-layer non-linear networks. Future work is needed to extend this analysis to end-to-end architectures and to provide a formal landscape proof for the rescaled MSE variant. These findings pave the way for designing more efficient loss functions grounded in geometric principles.
Summary updated