When humans evaluate choices, we rarely calculate the absolute, cumulative utility of an action in a vacuum. Instead, our internal feedback loop is deeply counterfactual: we look at what we chose, compare it to the best alternative we could have chosen, and experience a sense of opportunity cost. This psychological mechanism is known as regret.
Traditional behavioral models and standard Reinforcement Learning from Human Feedback (RLHF) assume that preferences are driven by maximizing absolute rewards. However, cognitive science and behavioral economics suggest otherwise. For instance, Li Li et al. demonstrate in Parsing cultural impacts on regret and risk in Iran, China and the United Kingdom that anticipated regret acts as a universal variable across cultures, where decision-makers actively seek to avoid outcomes where an unchosen option would have yielded a superior result.
If humans naturally evaluate choices through the lens of regret, forcing alignment algorithms to interpret human feedback as absolute rewards introduces a fundamental mismatch. A growing body of research argues that we should instead model human preferences directly as assessments of relative suboptimality—or regret.
The Mathematical Mismatch of Reward-Based RLHF
Why does it matter if we mistake regret for reward? In standard RLHF, a human evaluator is presented with two trajectories or actions and asked which one is better. The standard Bradley-Terry preference model assumes the human prefers trajectory $A$ over $B$ because the total reward of $A$ is higher:
$$P(A \succ B) = \sigma(R(A) - R(B))$$
However, W. Bradley Knox et al. reveal a fascinating consequence of this assumption in Learning Optimal Advantage from Preferences and Mistaking It for Reward. When human evaluators judge actions based on their regret—the gap between the chosen action and the optimal alternative—but the preference model assumes they are evaluating absolute rewards, the learned signal inadvertently converges to the optimal advantage function ($A^* = Q^* - V^*$).
Far from being a destructive error, this "mistake" explains why simple, bandit-style algorithms work surprisingly well for fine-tuning Large Language Models (LLMs): the advantage function acts as a highly shaped reward signal.
Explicitly embracing this regret-based perspective unlocks significant algorithmic advantages. As W. Bradley Knox et al. show in Models of human preference for learning reward functions, modeling preferences via regret ensures that the underlying reward function is mathematically identifiable. This resolves severe ambiguities that plague standard reward-summing models in environments with varying task lengths, allowing models to learn accurate reward functions with significantly less data.
Overcoming Sparsity and Environmental Noise
One of the most compelling reasons why regret-based preference modeling works better than reward maximization is how it handles credit assignment and sparse feedback.
Consider an environment where an agent fails a complex task. In a standard reward-based model, two failing trajectories might both receive a cumulative reward of zero, making them indistinguishable to the learning algorithm. A regret-based model, however, evaluates the gap between the achieved performance and the optimal policy from that specific state.
As Taehyun Cho et al. explain in Policy-labeled Preference Learning: Is Preference Enough for RLHF?, defining preferences through regret provides a much denser learning signal. By distributing feedback across timesteps based on how much an action deviates from the optimal path, the model can differentiate between a near-miss and a total catastrophic failure. Furthermore, the authors show that this formulation allows agents to disentangle whether a poor outcome was caused by the agent's suboptimality or by environmental stochasticity, leading to more robust offline and online learning.
Reward-Based Evaluation (Sparse):
Trajectory A (Near Miss): ───❌ Reward: 0
Trajectory B (Total Failure): ❌─── Reward: 0
(Result: Algorithm cannot distinguish quality)
Regret-Based Evaluation (Dense):
Trajectory A: ───❌ Regret: Low (Close to optimal path)
Trajectory B: ❌─── Regret: High (Far from optimal path)
(Result: Clear gradient for learning)
This capacity to handle suboptimality is leveraged by Joey Hejna et al. in Contrastive Preference Learning: Learning from Human Feedback without Reinforcement Learning. By defining human preferences as regret, their Contrastive Preference Learning (CPL) framework completely bypasses the unstable, two-stage reinforcement learning process. Instead, CPL converts preference learning into a stable, supervised contrastive learning task, making it highly effective for high-dimensional applications like robotics where rewards are inherently sparse.
From Bounded Rationality to LLM Optimization
The utility of regret minimization extends beyond reinforcement learning into structural econometrics and language modeling. From a foundational perspective, Silviu Pitis provides a decision-theoretic foundation for this in Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach, demonstrating that the error between an agent's true utility and a simplified fixed-discount model scales linearly with the agent's regret. This formalizes how much preference information is lost when we rely on rigid reward-discounting approximations instead of tracking regret directly.
In real-world environments, humans rarely achieve perfect Nash equilibria. Instead, they learn and err. In An Experimental Evaluation of Regret-Based Econometrics, Noam Nisan et al. demonstrate that assuming human agents merely minimize their regret—rather than acting with perfect static rationality—allows for highly robust estimation of hidden private valuations in auctions.
To better capture human bounded rationality, the authors introduce a "Quantal Regret" method in A "Quantal Regret" Method for Structural Econometrics in Repeated Games. By incorporating a regret aversion parameter, they model humans as probabilistically more likely to choose actions with lower regret, creating a flexible bridge between perfect rationality and purely random behavior.
This exact insight—that humans naturally evaluate "how bad" an outcome is relative to a hypothetical alternative—has fundamentally shifted how we approach LLM alignment. Suhwan Kim et al. introduce Regret-based Preference Optimization (RePO) in A Regret Minimization Framework on Preference Learning in Large Language Models. By reframing LLM training from reward maximization to a regret minimization objective, RePO explicitly accounts for the suboptimality of the data-generating policy. This counterfactual framing consistently outperforms standard baselines like Direct Preference Optimization (DPO) on both alignment and complex mathematical reasoning tasks.
Ultimately, as highlighted by Rui Yu et al. in Reward Models in Deep Reinforcement Learning: A Survey, moving toward regret-based frameworks allows reward models to mirror true human intent far more closely than absolute cumulative reward models ever could. Whether analyzing economic auctions, training robotic manipulators, or aligning language models, evaluating actions by the opportunity cost of the unchosen path provides a more stable, identifiable, and human-centric signal for learning.
Key Foundational Work
An Experimental Evaluation of Regret-Based Econometrics (2017) — contributed 1 concept elements to this direction.
A "Quantal Regret" Method for Structural Econometrics in Repeated Games (2017) — contributed 1 concept elements to this direction.
Parsing cultural impacts on regret and risk in Iran, China and the United Kingdom (2018) — contributed 1 concept elements to this direction.