Abstract
Several efforts have been done to bring ROC analysis beyond (binary) classification, especially in regression. However, the mapping and possibilities of these proposals do not correspond to what we expect from the analysis of operating conditions, dominance, hybrid methods, etc. In this paper we present a new representation of regression models in the so-called regression ROC (RROC) space. The basic idea is to represent over-estimation on the x axis and under-estimation on the y axis. The curves are just drawn by adjusting a shift, a constant that is added (or subtracted) to the predictions, and plays a similar role as a threshold in classification. From here, we develop the notions of optimal operating condition, convexity, dominance, and explore several evaluation metrics that can be shown graphically, such as the area over the RROC curve (AOC). In particular, we show a novel and significant result, the AOC is equal to the error variance (multiplied by a factor which does not depend on the model). The derivation of RROC curves with non-constant shifts and soft regression models, and the relation with cost plots is also discussed.
Introduction
In machine learning, regression models are typically evaluated using symmetric metrics such as Mean Squared Error (MSE) or Mean Absolute Error (MAE). These metrics operate under the assumption that over-estimating a value is as detrimental as under-estimating it. However, real-world scenarios often involve asymmetric costs. For instance, a retailer over-estimating demand faces inventory holding costs, while under-estimating leads to lost sales and customer dissatisfaction. While Receiver Operating Characteristic (ROC) analysis is the standard for navigating such trade-offs in binary classification, regression has lacked a comparable graphical framework. This paper introduces the Regression ROC (RROC) space to bridge this gap.
The RROC Space: A New Coordinate System
The author proposes a 2D representation where the performance of a regression model is defined by its total over-estimation and under-estimation. Specifically:
- X-axis (OVER): The sum of all positive errors (where the prediction is higher than the actual value).
- Y-axis (UNDER): The sum of all negative errors (where the prediction is lower than the actual value), typically plotted as negative values.
In this space, the "heaven point" is (0,0), representing a perfect model with zero error. Models closer to the origin are generally superior. To generate a curve rather than a single point, the concept of a shift ($s$) is introduced. By adding a constant $s$ to every prediction, a model's position moves through the RROC space, analogous to changing a classification threshold.
Theoretical Breakthrough: Linking AOC to Variance
The most significant contribution of this work is the mathematical link between visual performance and fundamental statistics. Theorem 7 in the paper proves that the Area Over the RROC Curve (AOC) is directly proportional to the model’s error variance:
$$AOC = \frac{\sigma^2 n^2}{2}$$
where $\sigma^2$ is the population variance of the errors and $n$ is the number of instances. This implies that while the position of the curve in the RROC space depends on the model's bias, the area it encloses is a pure measure of the model's dispersion (variance). For unbiased models, this area is directly related to the MSE.
Cost-Sensitivity and Model Selection
Operating conditions are represented by isometrics—lines of equal cost. The slope of these lines is determined by the cost asymmetry (the ratio of the cost of under-estimation to the cost of over-estimation).
By identifying the point on the RROC curve that is tangent to the isometric line closest to the origin, researchers can determine the optimal shift for any given cost scenario. Furthermore, the paper proves that RROC curves generated by constant shifts are always convex, allowing for the calculation of a convex hull to identify dominant models or optimal hybrid models.
Conclusion and Future Directions
The RROC framework provides a robust theoretical foundation for cost-sensitive regression. While the current work focuses on constant shifts and absolute error, future research may explore non-constant shifts (e.g., polynomial functions of predictions) or different loss functions like asymmetric squared error. By decoupling bias and variance visually, RROC curves offer a powerful new lens for model evaluation and optimization in complex, real-world regression tasks.
Summary updated