SLACE: A Monotone and Balance-Sensitive Loss Function for Ordinal Regression

Inbar Nachmani, Bar Genossar, Coral Scharf, Roee Shraga, Avigdor GalAAAI 2025 · DOI

TL;DR

This paper introduces SLACE, a novel loss function for ordinal regression that explicitly addresses the inherent order of labels and class imbalance. SLACE demonstrably outperforms existing state-of-the-art methods on various benchmarks by incorporating monotonicity and balance-sensitivity, leading to more accurate and robust ordinal classification.

What Problem Does This Paper Address?

Ordinal regression is a type of classification where the output labels have a natural order, such as risk levels (low, medium, high) or customer satisfaction ratings (poor, fair, good). While standard classification focuses solely on predicting the correct class, ordinal regression acknowledges that the severity of a misclassification depends on the distance between the predicted and true labels. For instance, misclassifying a "critical risk" as "low risk" is far worse than misclassifying it as "high risk." Existing ordinal regression loss functions often fail to adequately capture this nuanced error severity, either by not respecting the label order (monotonicity) or by not accounting for the varying importance of different classes due to their frequency (balance-sensitivity). This paper aims to bridge this gap by proposing a loss function that explicitly incorporates these crucial properties.

Methods

The core contribution of this paper is the SLACE (Soft Labels Accumulating Cross Entropy) loss function. SLACE is built upon two key desiderata for ordinal regression loss functions:

  • Monotonicity: The loss should penalize misclassifications further away from the true class more heavily. This is formalized as "e-ordinal monotonicity," meaning that if probability mass is shifted closer to the true class, the loss must decrease.
  • Balance-sensitivity: The loss should account for class imbalance. Errors involving rarer classes should incur a greater penalty because they represent a larger proportion of the overall information for that class.

SLACE achieves these properties through several mechanisms:

  • Proximity-Aware Closeness and Distance: Instead of assuming uniform distances between classes, SLACE uses a proximity measure that considers the relative frequency of classes to define distances. This allows the loss to adapt to the specific dataset's class distribution.
  • Accumulating Cross-Entropy (acc_loss): This intermediate loss function accumulates probabilities of classes, weighting them by their closeness to the true class. This ensures that classes further from the true label contribute less to the loss.
  • Soft Labels: SLACE further refines acc_loss by incorporating soft labels (derived from class probabilities and the proximity measure), which allows for a more nuanced weighting of class contributions.

The paper also proves that SLACE satisfies the Proper Scoring Rule (PSR) and is convex, which are desirable properties for optimization.

Datasets

The paper evaluates SLACE on a combination of datasets:

  • Inherently Ordinal Datasets: 10 datasets that naturally have ordered labels.
  • Binned Regression Datasets: 9 regression datasets that were modified by binning their continuous outputs into 5, 7, and 10 ordinal classes. This allowed for analysis of performance across varying numbers of classes.

The authors note that all datasets and code are available in an online repository.

Key Insights and Ideas

  • The importance of ordinality and balance: The paper strongly argues and empirically demonstrates that standard classification losses (like Cross-Entropy) and even existing ordinal losses are insufficient because they ignore the inherent order of labels and the impact of class imbalance.
  • Proximity matters more than uniform distance: The idea of using a dataset-specific proximity measure to define class distances is crucial. It allows the loss function to be more sensitive to the actual relationships between classes as represented in the data, rather than relying on a generic assumption.
  • SLACE's dual benefit: The proposed SLACE loss function effectively combines monotonicity and balance-sensitivity, providing a more holistic approach to ordinal regression that leads to improved performance.
  • Soft labels enhance ordinal losses: The integration of soft labels, particularly when combined with proximity-awareness, significantly benefits ordinal loss functions.

Key Results

  • Superior Performance: SLACEprox (SLACE with proximity-awareness) consistently outperformed state-of-the-art ordinal loss functions (CE, SORD, OLL) across multiple evaluation metrics (Accuracy, MAE, CEM, Kendall's Tau, QWK, Macro Accuracy, AMAE).
  • Robustness: SLACE's performance remained superior even when the number of classes increased, demonstrating its robustness.
  • Significant Improvements: For example, SLACE improved AMAE by up to 7.2% and Macro Accuracy by up to 5.4% compared to the best baseline (OLL) on ordinal and adjusted regression datasets, respectively.
  • Proximity-Awareness Benefits: The study showed that integrating proximity-awareness into soft-label-based loss functions (like SLACE) led to superior performance.

Limitations and Future Work

  • Tabular Data Focus: The empirical analysis primarily focused on tabular data. The authors suggest extending the investigation to include multiple data types, with an emphasis on multi-modal analysis.
  • Impact of Varying Distances and Sizes: The paper acknowledges that further investigation into how different class sizes and varying distances specifically impact model performance would be beneficial.

Summary updated

BibTeX
@inproceedings{nachmani2025slace,
  author = {Inbar Nachmani and Bar Genossar and Coral Scharf and Roee Shraga and Avigdor Gal},
  title = {{SLACE: A Monotone and Balance-Sensitive Loss Function for Ordinal Regression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i18.34158},
}
Figure