Regression as Classification: Influence of Task Formulation on Neural Network Features

Lawrence Stewart, Francis R. Bach, Quentin Berthet, Jean-Philippe Vert

Abstract

Neural networks can be trained to solve regression problems by using gradient-based methods to minimize the square loss. However, practitioners often prefer to reformulate regression as a classification problem, observing that training on the cross entropy loss results in better performance. By focusing on two-layer ReLU networks, which can be fully characterized by measures over their feature space, we explore how the implicit bias induced by gradient-based optimization could partly explain the above phenomenon. We provide theoretical evidence that the regression formulation yields a measure whose support can differ greatly from that for classification, in the case of one-dimensional data. Our proposed optimal supports correspond directly to the features learned by the input layer of the network. The different nature of these supports sheds light on possible optimization difficulties the square loss could encounter during training, and we present empirical results illustrating this phenomenon.

Introduction: The "Binning" Phenomenon

In modern machine learning, continuous targets—such as age estimation from images or 3D coordinate prediction in pose estimation—are traditionally handled via regression using a squared error loss. However, a persistent "trick" in the industry involves discretizing these continuous targets into discrete bins, effectively transforming the task into a classification problem optimized with cross-entropy loss.

Despite the fact that cross-entropy loss inherently ignores the spatial relationships between bins (treating all misclassifications equally regardless of distance), it frequently yields superior accuracy and more stable training. This paper investigates the "implicit bias" of gradient descent to explain why this counter-intuitive reformulation often outperforms standard regression.

Theoretical Framework

The authors analyze this phenomenon using two-layer ReLU neural networks within an "infinite-width" framework. By representing the network as a mathematical measure over its feature space, they can characterize the "kinks"—the specific points where ReLU activations trigger—as the primary features learned by the model.

The study focuses on comparing the "optimal supports" of these measures. In this context, the support represents the set of features (neuron activation points) that the optimizer prefers to utilize to minimize the respective loss functions.

Sparse vs. Dense Feature Support

The central theoretical insight is a stark difference in how regression and classification distribute their features. For regression tasks using squared loss, the optimal support ($R_{reg}$) is remarkably sparse. The network tends to place its features exclusively at the "kinks" or critical points of the target function.

In contrast, the classification formulation ($R_{class}$) favors a dense support. The cross-entropy loss encourages features to spread out across the entire input range. This density increases as the number of bins ($k$) grows, creating a "safety net" for optimization.

Comparison of optimal supports for regression ($R_{reg}$) and classification ($R_{class}$) on a simple absolute value function. Regression favors sparse features at the "kinks," while classification encourages dense features across the domain.

This difference has profound implications for optimization. Because the ideal regression features are so sparse, gradient descent may fail to find them if neurons are not initialized near these specific "kinks." This often leads to underfitting. Classification, by favoring a dense distribution, is much more robust to initialization; the optimizer is far more likely to find a functional set of features regardless of where they start.

Empirical Results

To validate these theoretical findings, the authors conducted experiments on synthetic 1D and 2D datasets. In a 1D "triangle" function test, the classification model (using 50 bins) achieved a Root Mean Square Error (RMSE) of $0.86 \times 10^{-2}$, significantly outperforming the best regression model's RMSE of $3.70 \times 10^{-2}$.

Beyond raw accuracy, classification demonstrated superior stability. As shown in the performance comparison, regression models exhibited high variance and frequently failed to capture fine-grained details of the target function.

Comparison of Root Mean Square Error (RMSE) for regression and classification tasks. Classification shows significantly lower error and higher stability across initializations.

Conclusion and Future Directions

While this work provides a rigorous explanation for the binning phenomenon in two-layer networks, several avenues for future research remain. The mathematical characterization of optimal support becomes increasingly complex in high-dimensional spaces, and extending these proofs to deep architectures is a significant challenge. Furthermore, while standard classification ignores bin adjacency, future work could investigate "soft" classification losses or Earth Mover's Distance to combine the dense feature bias of classification with the spatial awareness of regression.

Summary updated

Abstract

Neural networks can be trained to solve regression problems by using gradient-based methods to minimize the square loss. However, practitioners often prefer to reformulate regression as a classification problem, observing that training on the cross entropy loss results in better performance. By focusing on two-layer ReLU networks, which can be fully characterized by measures over their feature space, we explore how the implicit bias induced by gradient-based optimization could partly explain the above phenomenon. We provide theoretical evidence that the regression formulation yields a measure whose support can differ greatly from that for classification, in the case of one-dimensional data. Our proposed optimal supports correspond directly to the features learned by the input layer of the network. The different nature of these supports sheds light on possible optimization difficulties the square loss could encounter during training, and we present empirical results illustrating this phenomenon.
BibTeX
@inproceedings{stewart2023regression,
  author = {Lawrence Stewart and Francis R. Bach and Quentin Berthet and Jean-Philippe Vert},
  title = {{Regression as Classification: Influence of Task Formulation on Neural Network Features}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=WhEnuJyWRa},
}
Figure