MAPLE: A Framework for Active Preference Learning Guided by Large Language Models

Saaduddin Mahmud, Mason Nakamura, Shlomo ZilbersteinAAAI 2025 · DOI

TL;DR

This paper introduces MAPLE, a novel framework that uses Large Language Models (LLMs) to guide active preference learning. MAPLE significantly improves sample efficiency and human interaction by combining natural language feedback with traditional preference learning methods, leading to more accurate and interpretable learned preferences.

What Problem Does This Paper Address?

Aligning AI agents with human preferences is crucial for deploying them in real-world applications. Traditional methods for learning these preferences often rely on human-generated rankings of behaviors, which can be computationally expensive and require significant human supervision. While recent work has explored using LLMs to interpret natural language instructions for preference learning, these approaches often lack systematic query selection and can still be burdensome for humans. This paper addresses the need for a more efficient, interpretable, and human-friendly framework for preference learning that leverages the power of LLMs.

Methods

MAPLE builds upon Bayesian active preference learning by integrating LLMs in two key ways:

  • LLM-Guided Preference Modeling: MAPLE uses LLMs to interpret natural language instructions and estimate an initial distribution over preference functions. This distribution is then updated using both conventional preference feedback (like pairwise trajectory rankings) and linguistic feedback (explanations of preferences).
  • Language-Conditioned Active Query Selection: To reduce human burden, MAPLE employs a novel active query selection mechanism. This mechanism leverages feedback on the difficulty of previous queries to choose future queries that are both informative and easier for humans to answer. This is achieved through an "oracle-guided" approach where an LLM helps select the most suitable query.

The framework models preference functions as a linear combination of abstract language concepts, allowing for modularity and the acquisition of new concepts over time. This structure also facilitates human auditing of the learning process.

Datasets

The paper evaluates MAPLE on two benchmarks:

  • Minigrid (HomeGrid environment): A simplified setting designed to simulate robot household tasks.
  • OpenStreetMap Routing: A newly introduced, realistic benchmark for vehicle route planning using real-world road network data and text descriptions. This dataset includes concepts like "time," "speed," "safety," and "scenic views."

Key Insights and Ideas

  • Synergy of LLMs and Bayesian Learning: Combining LLMs for natural language understanding with Bayesian methods for uncertainty reduction proves to be more effective than using either alone.
  • Reducing Human Burden is Key: The language-conditioned active query selection significantly eases the interaction for humans by prioritizing easier-to-answer queries.
  • Interpretability through Concepts: Representing preferences as a linear combination of concepts makes the learned model more interpretable and auditable by humans.
  • Handling Ambiguity: The framework's ability to incorporate explanations alongside simple preference feedback helps in disambiguating preferences and improving learning.

Key Results

  • Sample Efficiency: MAPLE demonstrates superior sample efficiency compared to baseline methods like B-REX, especially when feedback is limited.
  • Preference Inference Quality: MAPLE achieves higher preference inference accuracy and better generalization to unseen trajectories.
  • OSM Routing Domain: MAPLE, using only linguistic feedback, outperforms B-REX with at least 10 instances of conventional feedback. The "Natural" language instructions were also effective.
  • HomeGrid Domain: MAPLE significantly outperforms B-REX, with models like Mistral-Large-2 achieving a substantial reduction in cost difference.
  • Impact of OAQS: The Oracle-Guided Active Query Selection (OAQS) strategy further improves performance, leading to higher accuracy and better query success rates compared to random or top-query selection strategies.

Limitations and Future Work

  • Ambiguous or Noisy Feedback: The paper acknowledges that MAPLE's performance can be constrained in scenarios with ambiguous or noisy user feedback.
  • More Complex Environments: Future work aims to extend MAPLE to more complex environments and tasks.
  • Diverse Linguistic Feedback: Exploring different types of linguistic feedback beyond simple explanations is a promising direction.
  • User Studies: Conducting user studies to evaluate the usability and effectiveness of MAPLE is suggested.
  • Multimodal Feedback: The authors are interested in leveraging multimodal feedback (e.g., text combined with visual or haptic cues) to enhance alignment.

Summary updated

BibTeX
@inproceedings{mahmud2025maple,
  author = {Saaduddin Mahmud and Mason Nakamura and Shlomo Zilberstein},
  title = {{MAPLE: A Framework for Active Preference Learning Guided by Large Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i26.34964},
}

Loading related content...

Figure