Ranking-aware Reinforcement Learning for Ordinal Ranking

Aiming Hao, Chen Zhu, Jiashu Zhu, Jiahong Wu, Xiangxiang ChuarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces Ranking-Aware Reinforcement Learning (RARL), a novel framework that unifies ordinal regression and Learning-to-Rank (L2R) using reinforcement learning. RARL employs a ranking-aware reward function to jointly optimize prediction accuracy and ranking consistency, and a Response Mutation Operation (RMO) to prevent training stagnation.

What Problem Does This Paper Address?

Ordinal regression and ranking tasks, common in areas like medical image analysis (e.g., disease severity) and content quality assessment (e.g., image aesthetics), involve predicting data with an inherent order. Traditional methods struggle to simultaneously capture both the precise value of an ordinal label (regression) and the correct relative ordering of items (ranking). Regression models often ignore ordinal dependencies, while classification approaches lose granularity. Existing Learning-to-Rank (L2R) methods, while focusing on relative order, often don't explicitly model the causal relationship between input signals and their ordinal structure within a unified framework. This paper aims to bridge this gap by developing a method that explicitly models these ordinal dependencies and jointly optimizes both regression and ranking objectives.

Methods

The core of RARL is a novel reinforcement learning framework that integrates regression and L2R. Instead of relying on learned reward models, RARL uses a ranking-aware verifiable reward function. This function has two main components:

  • Regression Reward ($R_{reg}$): A non-zero reward is given only if the predicted continuous value is within a specified tolerance ($\delta$) of the ground truth. This encourages precise value prediction.
  • Ranking Reward ($R_{rank}$): This is a composite reward that combines three aspects:
  • Length Compliance Reward ($r_{plen}$): Ensures the predicted sequence length matches the ground truth.
  • Ranking Consistency Reward ($r_{consis}$): Uses Kendall's Tau coefficient to enforce coherence between explicit ranking predictions and the implicit order derived from regression outputs.
  • Ranking Accuracy Reward ($r_{acc}$): Directly optimizes agreement with human annotations or reference standards using Kendall's Tau.

The final Ranking-Aware Reward ($R_{final}$) linearly combines the regression reward, ranking reward, and a format reward, with tunable weights ($\lambda_1, \lambda_2, \lambda_3$) to balance these objectives.

To address the common RL issue of entropy collapse (where the model converges to a suboptimal, deterministic policy and gradients vanish), RARL introduces the Response Mutation Operation (RMO). RMO probabilistically replaces a few low-reward responses in a training batch with ground-truth or high-reward reference answers. This injects artificial advantage variance, creating contrastive pairs and reactivating gradient flow to escape saddle points and encourage further exploration.

The training is conducted in a two-stage strategy: first, the model is trained on the regression task alone, and then the full ranking objective and RMO are introduced to jointly refine both capabilities.

Datasets

The paper evaluates RARL on three distinct benchmarks:

  • UTKFace: For facial age estimation.
  • COCO-REM: For object counting.
  • AVA: For aesthetic assessment.

Key Insights and Ideas

  • Unified Objective: The paper demonstrates that jointly optimizing regression and ranking within a single RL framework can lead to mutual improvement, outperforming methods that treat these tasks separately.
  • Verifiable Rewards: Using verifiable rewards, rather than learned ones, is crucial for tasks with clear correctness criteria, avoiding reward model bias.
  • RMO for Exploration: The Response Mutation Operation is a novel and effective technique to combat entropy collapse in RL, particularly useful for complex tasks with potentially noisy or ambiguous outputs.
  • Ranking-Centric Training: For subjective tasks like aesthetics, ranking-centric training alone can be more effective than relying solely on auxiliary regression supervision, suggesting a shift in how we approach such problems.

Key Results

RARL consistently and significantly outperforms baseline models and supervised fine-tuning (SFT) across all evaluated tasks and settings.

  • On UTKFace, RARL achieves a State-of-the-Art (SOTA) Mean Absolute Error (MAE) of 3.81 for the 7B model.
  • On AVA, RARL achieves a Spearman's Rank Correlation Coefficient (SRCC) of 0.803 for the 7B model.
  • The ranking performance (measured by Kendall's Tau, $\tau$) shows substantial gains, especially in multi-image scenarios, highlighting the effectiveness of the joint optimization.
  • Even on object counting, which isn't a typical ordinal task, RARL shows performance improvements due to its multi-image comparative reasoning capabilities.
  • Ablation studies show that the full RARL model, combining regression and ranking rewards, yields the best performance.
  • The two-stage training strategy and the inclusion of RMO consistently improve performance, with RMO reducing MAE from 4.17 to 4.02 in one experiment.

Limitations and Future Work

  • Entropy Collapse Mitigation: While RMO is effective, the paper notes that simultaneous optimization of all three rewards from the outset can lead to unstable training, necessitating the two-stage approach. Further research into more robust exploration strategies could be beneficial.
  • Hyperparameter Tuning: The effectiveness of the weighted combination of rewards ($\lambda_1, \lambda_2, \lambda_3$) and the RMO parameters (like $k$) might require careful tuning for different tasks.
  • Generalizability: While tested on three benchmarks, exploring RARL's performance on a wider range of ordinal tasks and domains would be valuable.
  • Future Directions: The paper suggests that combining VLMs with RLVR offers a principled pathway toward sample-efficient and interpretable ranking systems, hinting at broader applications of this approach. The success of ranking-centric training on subjective tasks also opens avenues for further exploration in unsupervised or weakly-supervised learning for ordinal tasks.

Summary updated

Abstract

Ordinal regression and ranking are challenging due to inherent ordinal dependencies that conventional methods struggle to model. We propose Ranking-Aware Reinforcement Learning (RARL), a novel RL framework that explicitly learns these relationships. At its core, RARL features a unified objective that synergistically integrates regression and Learning-to-Rank (L2R), enabling mutual improvement between the two tasks. This is driven by a ranking-aware verifiable reward that jointly assesses regression precision and ranking accuracy, facilitating direct model updates via policy optimization. To further enhance training, we introduce Response Mutation Operations (RMO), which inject controlled noise to improve exploration and prevent stagnation at saddle points. The effectiveness of RARL is validated through extensive experiments on three distinct benchmarks.
BibTeX
@misc{hao2026ranking,
  author = {Aiming Hao and Chen Zhu and Jiashu Zhu and Jiahong Wu and Xiangxiang Chu},
  title = {{Ranking-aware Reinforcement Learning for Ordinal Ranking}},
  year = {2026},
  eprint = {2601.20585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.20585},
}
Figure