Learning to Plan with Personalized Preferences

Manjie Xu, Xinyi Yang, Wei Liang, Chi Zhang, Yixin Zhu

Abstract

Effective integration of AI agents into daily life requires them to understand and adapt to individual human preferences, particularly in collaborative roles. Although recent studies on embodied intelligence have advanced significantly, they typically adopt generalized approaches that overlook personal preferences in planning. We address this limitation by developing agents that not only learn preferences from few demonstrations but also learn to adapt their planning strategies based on these preferences. Our research leverages the observation that preferences, though implicitly expressed through minimal demonstrations, can generalize across diverse planning scenarios. To systematically evaluate this hypothesis, we introduce Preference-based Planning (PbP) benchmark, an embodied benchmark featuring hundreds of diverse preferences spanning from atomic actions to complex sequences. Our evaluation of SOTA methods reveals that while symbol-based approaches show promise in scalability, significant challenges remain in learning to generate and execute plans that satisfy personalized preferences. We further demonstrate that incorporating learned preferences as intermediate representations in planning significantly improves the agent's ability to construct personalized plans. These findings establish preferences as a valuable abstraction layer for adaptive planning, opening new directions for research in preference-guided plan generation and execution.

Introduction: The Challenge of Ambiguity

Imagine asking a robotic assistant, "Help me prepare an apple." To a generic AI, this is a straightforward instruction-following task. However, for a human user, the "correct" execution is deeply personal: Should the apple be washed in the sink or a bowl? Should it be sliced or kept whole? Should it be served on a plate or a napkin?

Personalized Solution vs. Traditional Solution: Moving from explicit step-by-step instructions to implicit preference learning.

Current embodied AI systems often struggle with this ambiguity. They either adopt a "one-size-fits-all" strategy that ignores individual nuances or require exhaustive, step-by-step instructions that defeat the purpose of having an autonomous assistant. The core challenge lies in the sparsity of natural language, which often fails to capture the "how" and "where" of daily chores. This research addresses this gap by enabling agents to learn "hidden" preferences from just a few observations of a user’s past behavior, moving beyond generic instruction-following toward truly personalized assistance.

The Preference-based Planning (PbP) Benchmark

To systematically study this problem, we introduce the Preference-based Planning (PbP) benchmark. Built on NVIDIA Omniverse and OmniGibson, PbP features 290 unique preferences across 50 different domestic scenes, totaling 15,000 unique recordings.

The three-tiered hierarchy of preferences in the PbP benchmark, ranging from atomic actions to complex sequences.

The benchmark categorizes preferences into a three-tiered hierarchy to test agents across different levels of complexity:

  • Action Level: Fine-grained details, such as the specific volume of water to fill in a cup.
  • Option Level: Spatial and organizational choices, like storing specific tools in a particular drawer.
  • Sequence Level: Temporal ordering and prioritization, such as cleaning the kitchen environment before beginning to cook.

Unlike previous benchmarks that focus on navigation or explicit command-following, PbP specifically evaluates an agent's ability to resolve ambiguity by leveraging historical context and behavioral patterns.

Methodology: A Two-Stage Framework

We propose a two-stage learning-planning framework designed to bridge the gap between observation and execution:

  1. Few-Shot Preference Learning: The agent observes a small number of demonstrations (typically three) of a user performing a task. It must analyze these behavioral patterns to identify consistent preferences (e.g., "this user always puts fruit in the fridge").
  2. Preference-Guided Planning: Once the preference is inferred—either as a latent representation or an explicit text label—the agent uses it as a constraint to generate a sequence of actions for a new, unseen scenario.

A key insight from our work is that preferences serve as a vital intermediate representation. Treating planning as a direct mapping from pixels to actions (end-to-end) is computationally difficult and often inaccurate. By first predicting a "preference label," the subsequent planning process becomes significantly more robust.

Results and Discussion

Our evaluation of state-of-the-art (SOTA) models, including GPT-4V, LLaVA-Next, and Llama3, revealed several critical findings:

  • The Vision-Reasoning Gap: Vision-based models (e.g., LLaVA-Next) are competent at planning once a preference is known but struggle to infer that preference from raw video. Conversely, symbol-based models excel at the reasoning required to identify patterns in action sequences.
  • Two-Stage Success: Purely end-to-end models often failed, producing high Levenshtein distances (a measure of plan deviation) that suggested they were essentially guessing. In contrast, the two-stage approach using GPT-4.1 (Symbol-based) achieved the highest overall preference prediction accuracy at 79.6%.
  • Robustness of Symbolic Reasoning: Vision-based models tend to "anchor" preferences to specific visual features. If the scene layout changes, their accuracy drops. Symbol-based reasoning is more robust because it operates on abstract concepts (e.g., "Apple -> Fridge") that remain invariant across different environments.

Conclusion

This research establishes preferences as a valuable abstraction layer for adaptive planning in embodied AI. While symbol-based approaches show promise in scalability and reasoning, significant challenges remain in bridging the "vision gap"—improving the ability of multimodal models to extract abstract behavioral rules directly from video. Future work will focus on real-world data collection to capture the fluid and context-dependent nature of authentic human preferences, moving beyond synthetic primitives toward more natural human-robot collaboration.

Summary updated

Abstract

Effective integration of AI agents into daily life requires them to understand and adapt to individual human preferences, particularly in collaborative roles. Although recent studies on embodied intelligence have advanced significantly, they typically adopt generalized approaches that overlook personal preferences in planning. We address this limitation by developing agents that not only learn preferences from few demonstrations but also learn to adapt their planning strategies based on these preferences. Our research leverages the observation that preferences, though implicitly expressed through minimal demonstrations, can generalize across diverse planning scenarios. To systematically evaluate this hypothesis, we introduce Preference-based Planning (PbP) benchmark, an embodied benchmark featuring hundreds of diverse preferences spanning from atomic actions to complex sequences. Our evaluation of SOTA methods reveals that while symbol-based approaches show promise in scalability, significant challenges remain in learning to generate and execute plans that satisfy personalized preferences. We further demonstrate that incorporating learned preferences as intermediate representations in planning significantly improves the agent's ability to construct personalized plans. These findings establish preferences as a valuable abstraction layer for adaptive planning, opening new directions for research in preference-guided plan generation and execution.
BibTeX
@misc{xu2025learning,
  author = {Manjie Xu and Xinyi Yang and Wei Liang and Chi Zhang and Yixin Zhu},
  title = {{Learning to Plan with Personalized Preferences}},
  year = {2025},
  eprint = {2502.00858},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.00858},
}
Figure