Abstract
Neural networks can be trained to solve regression problems by using gradient-based methods to minimize the square loss. However, practitioners often prefer to reformulate regression as a classification problem, observing that training on the cross entropy loss results in better performance. By focusing on two-layer ReLU networks, which can be fully characterized by measures over their feature space, we explore how the implicit bias induced by gradient-based optimization could partly explain the above phenomenon. We provide theoretical evidence that the regression formulation yields a measure whose support can differ greatly from that for classification, in the case of one-dimensional data. Our proposed optimal supports correspond directly to the features learned by the input layer of the network. The different nature of these supports sheds light on possible optimization difficulties the square loss could encounter during training, and we present empirical results illustrating this phenomenon.
Introduction
In deep learning, there is a curious gap between theory and practice regarding task formulation. While regression—predicting continuous values—is traditionally solved using squared error loss, many state-of-the-art models for tasks like image colorization or pose estimation discretize the output into bins and treat it as a classification problem. This "binning phenomenon" often leads to superior performance, despite cross-entropy loss having no inherent concept of distance between bins.
This paper investigates whether the "implicit bias" of gradient-based optimization—the tendency of algorithms to prefer certain types of solutions—explains why classification is often easier for neural networks to learn than regression. The authors hypothesize that regression models seek a sparse set of features that are difficult to optimize, whereas classification models favor a denser feature set that is much more accessible during training.
Methods
The study focuses on two-layer ReLU neural networks analyzed in the "infinite-width" limit. In this regime, the network is represented as a mathematical measure over the space of possible features. The authors simplify the 1D feature space by focusing on "kinks" (points where the ReLU function transitions from zero to linear).
By deriving the "optimal supports" (the set of kink locations the optimization process naturally gravitates toward), the authors compare the requirements for a minimal-norm solution under square loss (regression) versus cross-entropy loss (classification). These theoretical findings are then tested on synthetic 1D and 2D datasets. The 1D dataset consists of 250 data points featuring triangular peaks of varying scales, while the 2D dataset uses 625 points on a grid.
Key Insights and Ideas
The core theoretical contribution is the identification of a fundamental difference in implicit bias:
- Sparse vs. Dense Bias: Regression exhibits a sparse implicit bias, attempting to place kinks only at specific points where the underlying function's slope changes. Classification, by contrast, has a denser bias, placing kinks at every boundary where the discrete label changes.
- The Optimization Trap: Because the "perfect" features for regression are so specific and sparse, gradient descent often fails to find them, leading to under-fitting. Since classification requires a much larger, more distributed set of features, the network is more likely to encounter the necessary components during the optimization process.
- The Role of Depth: Empirical tests suggest that three-layer networks do not suffer from this specific optimization difficulty as severely as two-layer networks, indicating that depth may help overcome the sparse support problem inherent in square loss.
Key Results
Empirical evaluations on synthetic tasks confirm the theoretical predictions:
- Performance Gap: In the 1D task, the classification model consistently outperformed regression. The best classification run achieved a validation RMSE of $0.86 \times 10^{-2}$, significantly better than the $3.70 \times 10^{-2}$ achieved by the best regression run.
- Stability: Classification proved far more stable across random initializations, with an RMSE standard deviation of $0.19$ compared to $1.38$ for regression.
- Visual Evidence: Plots of learned features reveal that regression models often "ignore" small-scale details because they fail to optimize the necessary sparse features.
- Higher Dimensions: 2D experiments corroborated these findings, showing that regression models seek sparse "lines" of support while classification models distribute features more evenly across the input space.
Limitations and Future Work
While the theoretical proofs are rigorous for two-layer networks, characterizing implicit bias in deeper networks remains an open challenge. Additionally, the analysis is most robust in low dimensions; the "curse of dimensionality" might alter these feature support interactions in high-dimensional spaces like raw pixels. Finally, since standard cross-entropy ignores the relationship between adjacent bins, future research could explore "soft" classification or losses that incorporate the ordered nature of binned data.
Summary updated