Cross Entropy versus Label Smoothing: A Neural Collapse Perspective

Li Guo, Keith W. Ross, Zifan Zhao, George Andriopoulos, Shuyang Ling, Yufeng Xu, Zixuan Dong

Abstract

Label smoothing loss is a widely adopted technique to mitigate overfitting in deep neural networks. This paper studies label smoothing from the perspective of Neural Collapse (NC), a powerful empirical and theoretical framework which characterizes model behavior during the terminal phase of training. We first show empirically that models trained with label smoothing converge faster to neural collapse solutions and attain a stronger level of neural collapse. Additionally, we show that at the same level of NC1, models under label smoothing loss exhibit intensified NC2. These findings provide valuable insights into the performance benefits and enhanced model calibration under label smoothing loss. We then leverage the unconstrained feature model to derive closed-form solutions for the global minimizers for both loss functions and further demonstrate that models under label smoothing have a lower conditioning number and, therefore, theoretically converge faster. Our study, combining empirical evidence and theoretical results, not only provides nuanced insights into the differences between label smoothing and cross-entropy losses, but also serves as an example of how the powerful neural collapse framework can be used to improve our understanding of DNNs.

Introduction

In deep learning, classifiers are typically trained using Cross-Entropy (CE) loss with one-hot labels. However, Label Smoothing (LS)—a technique that mixes target labels with a uniform distribution—is frequently employed to prevent overfitting and improve calibration. While LS is empirically successful, the underlying reasons for its superior convergence and geometric properties have remained unclear. This study investigates these properties through the lens of Neural Collapse (NC).

Neural Collapse describes four empirical regularities observed at the end of training: features of the same class collapse to their class mean (NC1); class means arrange themselves into a Simplex Equiangular Tight Frame (NC2); classifier weights align with these class means (NC3); and the classifier behaves like a nearest-class-mean rule (NC4). This paper explores how LS influences these dynamics and whether this explains its performance advantages.

Methodology

The authors employ a dual approach combining empirical observation with theoretical modeling. ResNet architectures were trained on CIFAR-10, CIFAR-100, STL-10, and Tiny ImageNet, with NC metrics tracked for up to 800 epochs. Theoretically, the study utilizes the Unconstrained Feature Model (UFM), treating last-layer features as free optimization variables to derive closed-form solutions for global minimizers. Furthermore, a second-order analysis of the optimization landscape was performed by calculating the condition number of the Hessian matrix at the global minimum.

Geometric Dynamics and Neural Collapse

A central finding is that LS acts as an accelerator for Neural Collapse. Models using LS reach a more pronounced state of collapse faster than those using standard CE. Furthermore, LS introduces a specific geometric bias. By assigning small, equal probabilities to non-target classes, it pushes features toward the symmetric Simplex ETF structure (NC2).

Scatter plots of NC1 vs. NC2 across different datasets, showing that Label Smoothing (LS) induces intensified NC2 at equivalent levels of NC1 compared to Cross-Entropy (CE).

As shown in the scatter plots, for any given level of feature collapse (NC1), LS models consistently exhibit stronger geometric symmetry (NC2) than CE models. This suggests that LS fundamentally reshapes the feature space toward the ideal NC configuration more effectively than standard cross-entropy.

Optimization Landscape and Convergence Speed

Empirical results across all tested datasets confirm that LS models exhibit a faster reduction in training error and NC metrics. For instance, on CIFAR-10, LS models stabilize their NC1 and NC2 levels significantly earlier than CE models.

Evolution of Error Rates and Neural Collapse metrics (NC1, NC2, NC3) over training epochs for various datasets. LS models consistently show faster convergence and lower NC values.

The theoretical analysis provides a mathematical foundation for these observations. The authors proved that the condition number of the Hessian at the global minimum is lower for LS. Since a lower condition number indicates a more "rounded" and less "stretched" loss landscape, it explains why optimizers navigate the LS landscape more efficiently, leading to faster convergence.

Discussion and Limitations

While LS generally improves model calibration by preventing overconfidence, the study identifies a "calibration paradox." Because LS induces a stronger level of NC1 (feature collapse), it can lead to worse calibration if post-processing techniques like temperature scaling are applied; the model becomes "too collapsed" for its own good in certain post-hoc scenarios.

Limitations of this work include the reliance on the Unconstrained Feature Model, which simplifies hierarchical feature learning, and the remaining open question regarding the direct causal link between specific NC levels and test-time generalization. Nonetheless, this research demonstrates that Neural Collapse is a powerful framework for understanding the fundamental differences between loss functions in deep learning.

Summary updated

Abstract

Label smoothing loss is a widely adopted technique to mitigate overfitting in deep neural networks. This paper studies label smoothing from the perspective of Neural Collapse (NC), a powerful empirical and theoretical framework which characterizes model behavior during the terminal phase of training. We first show empirically that models trained with label smoothing converge faster to neural collapse solutions and attain a stronger level of neural collapse. Additionally, we show that at the same level of NC1, models under label smoothing loss exhibit intensified NC2. These findings provide valuable insights into the performance benefits and enhanced model calibration under label smoothing loss. We then leverage the unconstrained feature model to derive closed-form solutions for the global minimizers for both loss functions and further demonstrate that models under label smoothing have a lower conditioning number and, therefore, theoretically converge faster. Our study, combining empirical evidence and theoretical results, not only provides nuanced insights into the differences between label smoothing and cross-entropy losses, but also serves as an example of how the powerful neural collapse framework can be used to improve our understanding of DNNs.
BibTeX
@article{guo2025cross,
  author = {Li Guo and Keith W. Ross and Zifan Zhao and George Andriopoulos and Shuyang Ling and Yufeng Xu and Zixuan Dong},
  title = {{Cross Entropy versus Label Smoothing: A Neural Collapse Perspective}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=FEo55EIvGI},
}
Figure