Regret-Based Preference Learning for Reinforcement Feedback

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 221 papers40 conceptsfrozen 2026-10-04

Standard Reinforcement Learning from Human Feedback (RLHF) relies on a simple assumption: when a human prefers one trajectory over another, they are mentally summing up the rewards along those paths and picking the one with the higher total. However, recent research suggests this "partial return" model is both mathematically flawed and cognitively inaccurate. Instead, a new perspective argues that human preferences are driven by regret—the gap between what an agent did and what it could have done.

The Failure of Cumulative Reward Models

In the traditional Bradley-Terry model used for RLHF, the probability that a human prefers trajectory $A$ over trajectory $B$ is proportional to the exponential of the difference in their accumulated rewards. W. Bradley Knox and colleagues have demonstrated that this model suffers from a "reward identifiability" problem Knox et al. (2022).

In many goal-oriented tasks, multiple reward functions can explain the same set of preferences but lead to wildly different optimal behaviors. This ambiguity is especially pronounced when trajectory segments vary in length or when the environment is stochastic. If we only look at the sum of rewards, we lose the context of the starting and ending states, making it impossible to distinguish whether an agent performed well or simply started in a "lucky" high-value state.

Regret as a Counterfactual Signal

To solve this, researchers propose modeling preferences through the lens of regret. In this framework, a human evaluator doesn't just look at the rewards earned; they perform a counterfactual evaluation: “How much value did the agent lose by not taking the optimal action?”

Formally, this regret is defined as the difference between the optimal value of a state and the value of the action actually taken. This aligns with formal logic systems like STIT ("Seeing To It That"), where Emiliano Lorini and François Schwarzentruber define counterfactual emotions as the intersection of a preference for an alternative state and the knowledge that a different choice could have reached it Lorini and Schwarzentruber (2009).

By evaluating the "opportunity cost" of actions, regret-based models capture the quality of decision-making relative to potential outcomes rather than just the immediate environment feedback Knox et al. (2022).

Learning the Advantage Function

A surprising discovery in this field is that the standard RLHF pipeline—used to train models like ChatGPT—actually works because it inadvertently treats feedback as regret. W. Bradley Knox, Anca D. Dragan, and Peter Stone argue that when we apply reward-summation algorithms to regret-based human feedback, we don't learn a raw reward function. Instead, we learn the optimal advantage function Knox et al. (2023).

The advantage function $A^*(s, a)$ represents the relative benefit of a specific action compared to the average action in that state. Because it is "pre-shaped," it provides a dense signal at every timestep, telling the agent exactly how much it is deviating from the optimal path. This explains why RLHF can effectively fine-tune models even in environments where rewards are sparse or terminal Knox et al. (2024).

Robustness and Disentanglement

Regret-based modeling offers two major practical advantages:

  1. Credit Assignment in Sparse Environments: In tasks where success only happens at the very end (like a robot successfully placing a cup), cumulative reward models struggle to assign credit to early actions. Regret-based models, such as Policy-labeled Preference Learning (PPL), distribute feedback more evenly across the trajectory by measuring the gap between achieved and optimal performance at each step Cho et al. (2025).
  2. Disentangling Skill from Luck: In stochastic environments, an agent might fail due to a "bad bounce" rather than a bad choice. Taehyun Cho and colleagues show that by modeling the regret of the behavior policy, agents can better distinguish between environmental randomness and actual suboptimality Cho et al. (2025).

Beyond Reinforcement Learning

The shift toward regret also enables new training paradigms that bypass reinforcement learning entirely. Contrastive Preference Learning (CPL), developed by Joey Hejna and Chelsea Finn, uses the regret framework to transform preference learning into a simple supervised contrastive task Hejna et al. (2024). By minimizing the regret of the preferred trajectory relative to the alternative, CPL achieves stable, scalable learning in high-dimensional robotics tasks without the instability of traditional RL.

This perspective even extends to human-AI interaction. Katherine M. Collins and Joshua B. Tenenbaum have modeled "tactful" explanations as a strategy to minimize a listener's counterfactual regret, suggesting that regret is not just a training signal, but a fundamental component of how humans communicate value and agency Collins et al. (2025).

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 11 papers
  2. Version 1 · 2026-10-04 · 21 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_regret_based_preference_modeling_88701939,
  title = {regret-based preference modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/regret-based-preference-modeling/rdr_887019394ad841a19d082c911d24858f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_regret_based_preference_learning_for_rei_88701939_v1,
  title = {Regret-Based Preference Learning for Reinforcement Feedback},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/regret-based-preference-learning-for-reinforcement-feedback/rdr_887019394ad841a19d082c911d24858f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}