Abstract
Policy selection is a critical challenge in offline reinforcement learning, particularly in high-stakes domains like autonomous driving and medical trials. Traditional Off-Policy Evaluation (OPE) methods attempt to estimate exact performance values, which often leads to high variance and inaccuracy. We present PoRank, a framework that reframes policy selection as a ranking problem. PoRank utilizes a Policy Comparison Transformer (PCT) to capture cross-policy discrepancies and a specialized Crowd Layer to aggregate noisy signals from multiple imperfect OPE algorithms. Our results on D4RL benchmarks demonstrate that PoRank significantly outperforms existing baselines in ranking accuracy and regret minimization.
Introduction
In fields such as autonomous driving, medical drug trials, or high-frequency trading, researchers often develop dozens of different "policies" (AI decision-making strategies). Before deploying one into the real world—where a mistake could be costly or dangerous—it is essential to identify the most effective candidate.
The standard approach, Off-Policy Evaluation (OPE), tries to estimate the exact numerical reward a policy would receive using historical data. However, OPE is notoriously difficult, often suffering from high variance or requiring complex, hard-to-build environment models. PoRank shifts the goal from absolute estimation to relative ranking. In most practical scenarios, knowing that Policy A is better than Policy B is more valuable than knowing their exact predicted rewards.
Methodology
PoRank consists of two primary components designed to improve the robustness of policy selection.
1. Policy Comparison Transformer (PCT)
Unlike standard models that represent policies in isolation, the PCT focuses on pairwise comparisons. It takes a set of states from a dataset and evaluates the actions both policies would take. By encoding these side-by-side decisions into a Transformer, the model identifies the "cross-policy" discrepancies that are most indicative of performance differences.
2. Learning-to-Rank (LTR) with a Crowd Layer
A major hurdle in training a ranker is the lack of "ground truth" labels, as true rankings are unknown until policies are deployed. PoRank addresses this by treating existing, imperfect OPE algorithms as "crowd workers." A specialized Crowd Layer (CL) learns the reliability and bias of each OPE method. If an OPE method is consistently noisy, the Crowd Layer down-weights its contribution, allowing the system to converge on a more accurate ranking than any single algorithm could provide alone.
Experimental Setup
The framework was evaluated using the D4RL dataset, a standard benchmark for offline reinforcement learning, focusing on MuJoCo physics simulations (HalfCheetah-v2 and Walker2d-v2). The researchers tested across various trajectory types—including "expert," "medium," and "full-replay"—to assess performance across different data quality levels. Two distinct sets of candidate policies were used: one trained with diverse offline RL algorithms and another with online SAC to test generalization capabilities.
Key Results and Insights
- Superior Ranking Accuracy: PoRank achieved the highest Spearman’s Rank Correlation in 11 out of 12 tested scenarios, demonstrating a superior ability to match predicted ranks with true performance.
- Regret Minimization: In the "Regret@1" metric, PoRank consistently outperformed baselines, frequently identifying the absolute best policy or a near-optimal alternative.
- Resilience to Noise: The Crowd Layer effectively filtered out algorithmic noise, maintaining high performance even when the provided OPE labels were of low quality.
- Generalization: By focusing on decision-level differences rather than raw trajectory data, the PCT generalizes effectively to unseen policies not present in the training set.
Conclusion and Future Work
PoRank offers a more practical and robust alternative to traditional OPE by focusing on the relative ranking of policies. While the framework shows significant promise, it does involve computational overhead from running multiple OPE methods for training. Future research may explore combining PoRank with active learning or more advanced crowdsourcing techniques to reduce the initial data requirements while maintaining high ranking precision.
Summary updated