Active Preference Learning with Discrete Choice Data

Eric Brochu, Nando de Freitas, Abhijeet GhoshNeurIPS 2007 · OpenReview

Introduction

In creative fields like computer graphics and animation, artists often have to "twiddle" sliders—adjusting parameters like specularity, Fresnel coefficients, or motion gait—to achieve a specific look. This process is tedious because the mapping is non-intuitive, the "objective function" is entirely internal to the artist, and human rating is notoriously unreliable due to scale drift over time.

This paper introduces an active learning framework that helps users find their ideal settings in complex parameter spaces using only simple pairwise comparisons. By focusing on finding the maximum of a hidden preference function rather than modeling the entire space, the algorithm significantly reduces the number of trials needed to reach a satisfying result.

Methods: The Active Preference Learning Loop

The authors propose an active preference learning loop designed to offload cognitive burden from the human to an active learning agent. It consists of several key steps:

  1. Discrete Choice Data: Instead of asking for an absolute numerical score, the system presents two options and asks the user: "Which do you prefer?" This exploits the human strength in comparative judgment.
  2. Valuation Modeling: The system uses Gaussian Processes (GPs) to model the user's underlying (latent) valuation function. It employs a Thurstone-Mosteller model (a probit regression approach) to link these continuous latent values to the discrete binary choices provided by the user.
  3. Inference: Because the posterior distribution isn't analytically tractable, Laplace approximation is used to efficiently estimate the latent utility function.
  4. Expected Improvement (EI): Instead of picking points where the model is most uncertain (standard active learning), the algorithm picks points that maximize the Expected Improvement over the current best-known point. This balances exploration (trying new areas) and exploitation (honing in on high-value areas).
  5. Optimization: To find the next query point, the system optimizes the EI function using the DIRECT algorithm, a derivative-free global optimization technique.

Key Insights: Optimization vs. Regression

The core insight of the paper is that the system does not need to model the entire preference surface accurately; it only needs to be accurate near the peak.

Comparison of regression and optimization models

Traditional active learning often wastes queries trying to resolve uncertainty in "boring" or low-value regions of the parameter space. By treating preference as a latent continuous function and focusing purely on optimization, the researchers can apply powerful Bayesian optimization tools to subjective artistic taste while minimizing user fatigue.

Datasets and Results

The algorithm was tested on synthetic mathematical functions (2D, 4D, and 6D) designed to mimic complex landscapes with multiple local maxima, as well as a practical application using a subset of 38 measured materials from the MERL BRDF database.

Synthetic Efficiency: In 2D and 4D tests, the Expected Improvement (EI) approach reached the global maximum significantly faster than a "maximum variance" standard active learning baseline.

Performance across 2D, 4D, and 6D functions

User Study: In a practical study where users had to find a specific target material in a "Preference Gallery", the proposed EI method required only ~8.6 clicks on average. This was a massive improvement over Random/Latin Hypercubes (~18.4 clicks) and Maximum Variance (~17.9 clicks). Qualitatively, the predictor correctly identified the region of the global maximum even when it had a very poor understanding of the rest of the function's landscape.

Limitations and Future Work

While highly effective in lower dimensions, the method faces the "Curse of Dimensionality." As seen in the 6D synthetic tests, the search space becomes so vast that 50 queries (a reasonable limit for human patience) are not enough to explore it adequately.

For future work, the authors suggest that in higher dimensions, the system might need to allow users to "nudge" the search in specific directions rather than relying purely on binary choices. Additionally, while GPs can become slow as data points grow, this is less of an issue here since the primary goal is to keep the number of user clicks strictly bounded.

Summary updated

Abstract

We propose an active learning algorithm that learns a continuous valuation model from discrete preferences. The algorithm automatically decides what items are best presented to an individual in order to find the item that they value highly in as few trials as possible, and exploits quirks of human psychology to minimize time and cognitive burden. To do this, our algorithm maximizes the expected improvement at each query without accurately modelling the entire valuation surface, which would be needlessly expensive. The problem is particularly difficult because the space of choices is infinite. We demonstrate the effectiveness of the new algorithm compared to related active learning methods. We also embed the algorithm within a decision making tool for assisting digital artists in rendering materials. The tool finds the best parameters while minimizing the number of queries.
BibTeX
@inproceedings{brochu2007active,
  author = {Eric Brochu and Nando de Freitas and Abhijeet Ghosh},
  title = {{Active Preference Learning with Discrete Choice Data}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2007},
  url = {https://openreview.net/forum?id=ry-S8PWO-S},
}
Figure