Learning and Optimization from Relative Preference Feedback

191 concepts136 papers2 proposals

In traditional machine learning and optimization, we assume the environment provides absolute scalar feedback. An agent takes an action and receives a reward of exactly $+2.5$; a computer vision model generates an image and receives a loss score; a clinical patient evaluates their pain on a strict scale from 1 to 10.

However, in systems that involve human judgment, this assumption breaks down. Humans are notoriously poor at providing consistent absolute scores. If you ask a user to rate a generated image or a robotic exoskeleton's walking gait on a numerical scale, the feedback will be highly noisy, inconsistent across sessions, and heavily biased by context.

Conversely, humans excel at making relative comparisons. If you present a user with two options—Image A versus Image B, or Gait A versus Gait B—they can reliably tell you which one is better. This realization has driven a powerful paradigm shift: learning and optimizing systems using relative preference feedback. Instead of trying to force human intuition into absolute numbers, these frameworks design algorithms that natively learn from pairwise duels, rankings, and relative constraints.

The Cognitive and Mathematical Case for Relativity

The core motivation for relative preference feedback is rooted in human psychology. As William W. Cohen et al. demonstrate in Learning to Order Things, observing relative preferences provides a more natural and accessible source of training data than absolute relevance scores. This approach allows systems to learn effectively from implicit signals, such as a user choosing one search result over another.

Eric Heim leverages this in Constrained Generative Adversarial Networks for Interactive Image Generation, showing that because humans struggle to provide absolute numerical values to describe a desired visual outcome, interactive generative tasks become significantly more effective when guided by comparative constraints (e.g., "Generate an image more like Image A than Image B"). Similarly, Guangyi Zhang et al. note in Finding Favourite Tuples on Data Streams with Provably Few Comparisons that while absolute utility elicitation is highly unreliable for identifying a user's favorite item in a complex dataset, users can easily and consistently identify a preference when presented with a pair.

Mathematically, swapping absolute rewards for relative comparisons might seem like a downgrade that throws away valuable information. However, optimization theory proves otherwise. Kevin Jamieson et al. show in Query Complexity of Derivative-Free Optimization that while noisy derivative-free optimization is fundamentally slower than gradient-based methods, algorithms relying only on noisy "better-or-worse" comparisons can achieve the exact same optimal convergence rates as those using full, exact numerical evaluations. In short, you do not need absolute numbers to find the global optimum; relative directions are mathematically sufficient.

The Dueling Bandit Framework

To formalize this mathematically, researchers developed the Dueling Bandit problem. In a standard multi-armed bandit, an algorithm chooses an arm $i$ and observes a scalar reward $r_i$. In a dueling bandit, introduced by Yisong Yue et al. in The K-armed Dueling Bandits Problem, the algorithm selects a pair of arms $(i, j)$ and observes the outcome of a probabilistic duel:

$$P(i \succ j) = \frac{1}{2} + \epsilon_{ij}$$

where $i \succ j$ denotes that arm $i$ beat arm $j$, and $\epsilon_{ij}$ represents the relative advantage of $i$ over $j$. The goal is to find the optimal choice (the Condorcet winner, which beats all other options with a probability greater than 0.5) while minimizing "regret"—the cost of choosing suboptimal pairs during learning.

As Yanan Sui et al. explain in Advancements in Dueling Bandits, the core algorithmic challenge is that the absolute utility of an action is never directly observed. Algorithms must infer the optimal choice entirely from the relative win probabilities. To bridge this gap, algorithms like the Relative Minimum Empirical Divergence (RMED) framework, proposed by Junpei Komiyama et al. in Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problem, use information-theoretic divergence measures to optimally balance exploring uncertain pairs and exploiting known winners.

Reconstructing Latent Utility Spaces

How do these algorithms navigate continuous or high-dimensional spaces when they only receive binary "win/loss" signals? They do so by modeling a latent utility function that explains the observed pairwise preferences.

The standard approach maps a continuous parameter space (like the settings of a medical device or the weights of a neural network) to a latent reward space using probabilistic choice models, such as the Bradley-Terry-Luce (BTL) or Plackett-Luce models. Under these models, the probability that option $x$ is preferred over option $y$ is a function of their latent utilities $f(x)$ and $f(y)$:

$$P(x \succ y) = \sigma(f(x) - f(y))$$

where $\sigma$ is a sigmoid function. By modeling $f(x)$ as a shared underlying landscape, every pairwise comparison provides a piece of structural evidence about the slope of $f(x)$.

Pairwise Comparison (A vs B) ──> Sigmoid Link Function ──> Latent Utility Landscape f(x)
       (User prefers A)             P(A ≻ B) = σ(f(A)-f(B))        (Updated via GP or NN)

This latent utility can be modeled using different function approximators depending on the complexity of the problem:

  1. Gaussian Processes: In Preferential Bayesian Optimization, Javier González et al. model the latent utility function $f(x)$ as a Gaussian Process (GP). Even though they never observe $f(x)$ directly, they can update the GP's posterior using the likelihood of the observed pairwise duels. To scale this to large or continuous action spaces, Yanan Sui et al. demonstrate in Multi-dueling Bandits with Dependent Arms that GPs can model the probability of an item defeating a theoretical optimal environment, effectively bypassing the need for absolute utility observations.
  2. Neural Networks: When the preference landscape is highly non-linear, linear or kernel-based models fall short. Arun Verma et al. introduce Neural Dueling Bandits (NDB) in Neural Dueling Bandits: Preference-Based Optimization with Human Feedback, utilizing deep neural networks to approximate complex human preferences from pairwise comparisons while maintaining theoretical sub-linear regret guarantees.

Real-World Applications and Alignment

The power of learning from relative preferences is most evident in complex, high-stakes, or subjective domains where engineering a mathematical reward function is practically impossible.

Physical and Clinical Human-in-the-Loop Systems

In biomedical engineering, defining what makes a gait "comfortable" or a therapy "effective" cannot be reduced to a simple formula. In Human Preference-Based Learning for High-dimensional Optimization of Exoskeleton Walking Gaits, Maegan Tucker et al. developed the LINECOSPAR framework to personalize robotic exoskeleton gaits. By breaking down a high-dimensional optimization space into a series of simple 1D pairwise choices presented to the user, the system identifies ideal walking styles in remarkably few trials.

Similarly, Yanan Sui et al. apply this to personalized medicine in Correlational Dueling Bandits with Application to Clinical Treatment in Large Decision Spaces. Their algorithm, CORRDUEL, optimizes spinal cord stimulation settings for paralyzed patients using noisy, pairwise clinical feedback, matching or exceeding the performance of expert clinicians without ever requiring an absolute performance metric.

Reinforcement Learning and LLM Alignment

In modern artificial intelligence, relative preference feedback is the cornerstone of aligning models with human intent. As reviewed by Youssef Abdelkareem et al. in Advances in Preference-based Reinforcement Learning: A Review, Preference-based Reinforcement Learning (PbRL) eliminates the need for complex reward engineering—which frequently suffers from "reward hacking," where agents exploit loopholes in numerical reward functions. Instead, training agents via relative preferences forces them to align with actual human intuition.

This paradigm extends directly to text generation and prompt engineering. Arun Verma proposes the APOHF framework in Prompt Optimization with Human Feedback, which optimizes prompts for black-box Large Language Models (LLMs). Rather than requiring a validation dataset or a complex scoring metric, the system uses a dueling bandit strategy to iteratively refine prompts based entirely on simple "A vs. B" human preferences.

By shifting the foundational data primitive from an absolute scalar to a relative comparison, preference-based learning bridges the gap between rigid algorithmic optimization and the nuanced, qualitative nature of human judgment.

Key Foundational Work

Learning to Order Things (1997) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 149 papers
  2. Version 1 · 2026-10-04 · 132 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_relative_preference_feedback_6451434e,
  title = {relative preference feedback},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-preference-feedback/rdr_6451434e757e44a18e3720f6c5feb98b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_relative_preference_feedback_6451434e_v2,
  title = {relative preference feedback},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-preference-feedback/rdr_6451434e757e44a18e3720f6c5feb98b/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake