Introduction
In modern artificial intelligence, particularly in the development of Large Language Models (LLMs) and robotics, "reward engineering"—the manual specification of a mathematical reward function—has become a significant bottleneck. Manually defined rewards are often prone to "reward hacking," where an agent exploits flaws in the function to achieve a high score without performing the intended task. To circumvent this, researchers have turned to reward learning, where models are trained based on human feedback, such as preferences between two different policy outcomes.
While empirically successful, reward learning has lacked a rigorous theoretical foundation. Specifically, the field has lacked clear answers regarding optimal query design (which policies to show a human) and scaling laws (how the complexity of policy and reward models affects data requirements). This paper addresses these gaps by framing reward learning as a doubly nonparametric bandit problem, where both the reward and policy spaces are treated as infinite-dimensional function classes.
The Doubly Nonparametric Framework
Existing theories in bandit learning typically consider "singly nonparametric" cases, where either the reward or the policy space is complex, but not both. However, modern AI involves complex neural networks for both the policy and the reward model.
To model this reality, the authors represent rewards ($r$) and policies ($\pi$) as functions belonging to Reproducing Kernel Hilbert Spaces (RKHS). This allows for the handling of infinite-dimensional spaces while maintaining mathematical tractability. The interaction between a policy and a reward is modeled via a Linear Evaluation Functional:
$$F(\pi, r) = \langle r, M\pi \rangle$$
Here, $M$ is an operator that maps the policy into the reward space. The learner's goal is to estimate $r$ from noisy human feedback and then identify a policy $\pi$ that maximizes the estimated reward.
Methodology: Passive Querying and Optimal Design
The authors propose a strategy based on a Ridge Regression Plug-in Estimator. The learner first collects a set of queries (policies), observes noisy feedback, and uses ridge regression to estimate the reward function. This estimate is then "plugged" into a policy optimizer to select the final action.
Crucially, the paper focuses on Optimal Experimental Design. Rather than querying randomly or adaptively, the authors solve an optimization problem to identify a query set $Q$ that minimizes the "excess risk"—the gap between the performance of the optimal policy and the policy selected by the learner. This selection is guided by the spectral analysis of the operators defining the RKHS. The efficiency of learning is shown to be governed by the eigenvalue spectrum of these operators; essentially, how quickly the eigenvalues decay serves as a proxy for the "effective dimensionality" of the task.
Key Insights and Scaling Laws
1. Passive vs. Active Querying
One of the paper's most surprising findings is that a carefully designed passive querying strategy (where all queries are chosen at once) can theoretically outperform popular active/adaptive strategies like GP-UCB (Gaussian Process Upper Confidence Bound). Adaptive methods often rely on confidence intervals that can become loose or lead to detrimental feedback loops in high-dimensional spaces. The authors demonstrate that their passive approach avoids these pitfalls while achieving superior theoretical rates.
2. Dimension-Free Learning
The derived bounds suggest that reward learning can remain effective even as the nominal dimension ($d$) of the space approaches infinity. As long as the "effective dimension" (determined by the smoothness of the kernels and the eigenvalue decay) remains manageable, the learner can converge on an optimal policy efficiently.
3. Scaling Law Intuition
The authors derive a theoretical rate for excess risk. For a power-law eigenvalue decay with exponent $\beta$, the algorithm achieves an excess risk of $O(n^{-\frac{\beta}{\beta+2}})$. This provides a concrete scaling law: the quality of the learned policy improves predictably as the number of queries ($n$) increases, provided the policy class grows at a slower rate relative to the reward class.
Results and Empirical Validation
Beyond theoretical derivations, the paper provides evidence through specialized applications and simulations:
- Kernel Multi-Armed Bandits (MAB): When applied to the Kernel MAB problem using Matérn kernels, the proposed algorithm achieves a regret bound of $\tilde{O}(T^{\frac{4\nu+d(4d+6)}{6\nu+d(4d+7)}})$. This bound is strictly better than those currently established for standard GP-UCB and GP-Thompson Sampling.
- Simulation Study: In synthetic simulations using $d$-dimensional RKHSs, the predicted theoretical decay rate of $O(n^{-0.27})$ almost perfectly matched the observed empirical rate of $O(n^{-0.28})$. This close alignment suggests that the derived upper bounds may be minimax optimal.
Conclusion and Future Work
This work provides the first comprehensive theoretical framework for reward learning in complex, nonparametric environments. By establishing scaling laws and demonstrating the power of optimal passive design, it offers a roadmap for more data-efficient AI training.
However, limitations remain. The current theory relies on regularity assumptions regarding the alignment of the policy set with the reward mapping's eigenvectors. Future research is needed to explore "closed-loop" dynamics, where policy learning and reward querying occur simultaneously, and to investigate whether an optimal active querying strategy can be developed to further improve upon these passive rates.
Summary updated