Abstract
Ordinal regression with anchored reference samples (ORARS) has been proposed for predicting the subjective Mean Opinion Score (MOS) of input stimuli automatically. The ORARS addresses the MOS prediction problem by pairing a test sample with each of the pre-scored anchored reference samples. A trained binary classifier is then used to predict which sample, test or anchor, is better statistically. Posteriors of the binary preference decision are then used to predict the MOS of the test sample. In this paper, rigorous framework, analysis, and experiments to demonstrate that ORARS are advantageous over simple regressions are presented. The contributions of this work are: 1) Show that traditional regression can be reformulated into multiple preference tests to yield a better performance, which is confirmed with simulations experimentally; 2) Generalize ORARS to other regression problems and verify its effectiveness; 3) Provide some prerequisite conditions which can insure proper application of ORARS.
Introduction
In many fields, such as evaluating speech quality (Mean Opinion Score or MOS) or medical assessments, the goal is to predict a continuous numerical value. Traditional regression models attempt to map input features directly to these absolute scores. However, absolute scores are notoriously noisy. For a human judge, it is significantly more difficult to assign a precise score (e.g., "3.8") than it is to make a relative comparison (e.g., "Recording A sounds better than Recording B").
This paper demonstrates that treating continuous regression as a series of binary preference tests significantly improves prediction accuracy. By utilizing a framework called Ordinal Regression with Anchored Reference Samples (ORARS), the authors show that relative comparisons are more robust than absolute scoring, outperforming traditional regression on 15 out of 16 benchmark datasets.
The ORARS Framework
The ORARS framework formalizes the preference-based approach into two distinct stages:
- Binary Classifier Training: Instead of training a model to output a direct score, a binary classifier $g(x_t, x_a)$ is trained. This model takes a pair of samples—a test sample and an "anchor" sample with a known score—and predicts the probability that the test sample is "better" (has a higher score) than the anchor.
- Scoring with Preference: To determine a final numerical score for a new sample, the system compares it against a set of "anchored reference samples" (typically the training set). The final score is derived from the classifier's win/loss record across these comparisons.
To isolate the benefits of the scoring mechanism itself, the authors also introduce sORARS (simplified ORARS). This variant uses a standard regression model to make the "A vs B" decisions, allowing researchers to determine if improvements stem from the scoring logic or the binary training process.
Theoretical and Experimental Insights
The core advantage of ORARS lies in its ability to "smooth out" the distribution of errors inherent in standard regressors. Even if a regressor possesses a systematic bias, relative comparisons help cancel out some of that noise.
The authors provide a mathematical proof showing that if a regressor's error follows a uniform distribution, the preference-based scoring framework reduces the Mean Absolute Error (MAE) by at least $M/6$ (where $M$ is the error bound). Furthermore, experimental results suggest that training a model specifically for pairwise comparisons (ORARS) is superior to using a pre-trained regressor for those comparisons (sORARS). This indicates that the "preference" signal is a more stable target for machine learning than the "absolute score" signal.
Evaluation and Results
The researchers evaluated ORARS on 16 regression benchmark datasets covering medical, engineering, economic, and technical domains. The datasets ranged from 43 to over 22,000 samples.
Key findings include:
- Broad Superiority: ORARS achieved better performance than the baseline General Neural Network Regression (GRNN) on 15 out of 16 datasets.
- Significant Gains: In datasets like Census Domains and California Housing, the error reduction was substantial (e.g., Census Domains(1) MAE dropped from 22,708 to 18,466).
- Scoring vs. Training: While sORARS outperformed the baseline on 9/16 tasks, the full ORARS (with binary training) was the clear winner, beating sORARS on 12/16 tasks.
Conclusion and Future Work
While ORARS offers significant accuracy improvements by mimicking human psychometric logic, it introduces computational challenges. Training on pairs increases complexity to $O(N^2)$, and inference requires multiple comparisons against anchors. Future work will focus on identifying representative subsets of anchors to reduce this overhead and resolving potential "circular preferences" in ranking optimizations.
Summary updated