Learning to Plan with Personalized Preferences

Manjie Xu, Xinyi Yang, Wei Liang, Chi Zhang, Yixin ZhuSubmission to ICLR 2026 · Withdrawn

TL;DR

When assisting humans in their homes, embodied AI agents shouldn’t need tedious step-by-step instructions for everyday tasks—they should infer personal habits from just a few observations. This paper introduces the Preference-based Planning (PBP) benchmark and demonstrates that framing personalized planning as a two-stage pipeline (first predicting an explicit preference representation, then planning conditioned on it) dramatically outperforms end-to-end multimodal planning.

What Problem Does This Paper Address?

Natural language commands like "Help me prepare an apple" are inherently underspecified. Does the user like their apple peeled? Sliced? Washed in the sink or a bowl? Placed on a specific plate?

Current embodied AI systems typically either assume universal, "one-size-fits-all" commonsense behaviors or require exhaustive, repetitive back-and-forth prompting. In reality, human preferences are personalized, often subconscious, and rarely articulated verbally. However, humans can easily infer preferences from watching just one or two demonstrations.

This paper tackles few-shot preference learning and preference-guided embodied planning: enabling an agent to watch a handful of user demonstrations, infer the underlying personal preference, and immediately adapt its action plans in novel environments and setups.

Methods

The authors formulate the task as few-shot learning from demonstration and systematically compare two paradigms:

  1. End-to-End Planning: Feeding few-shot demonstration trajectories (egocentric video or symbolic action logs) directly into a foundation model alongside the current state, prompting it to generate the full personalized action plan in one pass.
  2. Two-Stage Learning-and-Planning:
  3. Stage 1 (Preference Inference): The agent abstracts the user’s implicit behavior into an explicit preference representation ($p = f(O)$).
  4. Stage 2 (Conditioned Planning): The agent uses the inferred preference token alongside the current scene state to generate a customized sequence of primitive actions ($a \sim g(s, p)$).

The authors evaluate both vision-based models (ViViT, LLaVA-NeXT, EILEV, GPT-4V) and symbol-based models that receive symbolic action logs (DAG-Opt / NOTEARS, Llama-3-8B, DeepSeek-R1, GPT-4.1).

Datasets

The authors introduce the Preference-based Planning (PBP) benchmark:

  • Simulation Environment: Built atop NVIDIA Omniverse and OmniGibson across 50 detailed indoor scenes.
  • Preference Hierarchy: 290 unique preferences structured into three tiers:
  • Action Level (75 preferences): Fine-grained execution parameters (e.g., water fill amount, cooking doneness).
  • Option Level (135 preferences): Alternative approaches to sub-tasks (e.g., storing fruit in the fridge vs. on the table).
  • Sequence Level (80 preferences): High-level task prioritization and order constraints (e.g., cleaning before cooking vs. after).
  • Scale: A reusable pool of 15,000 demonstration recordings (egocentric video, third-person views, bird's-eye maps, and frame-level annotations across 17 action primitives) with a standardized 5,000-instance few-shot test set.

Key Insights and Ideas

  • End-to-End Planning Fails Dramatically: State-of-the-art vision-language models struggle to map demonstration videos straight to execution plans; their action sequence errors (Levenshtein distances) hover close to the length of the ground-truth sequences themselves. Models memorize isolated low-level actions rather than discovering cohesive goals.
  • Preferences Serve as a Crucial Abstraction Layer: Decomposing the problem into explicit preference classification followed by preference-conditioned planning yields massive gains across every model family.
  • Vision-Language Models Have an Abstraction Bottleneck: Vision models like GPT-4V and LLaVA-NeXT struggle to infer preferences from video observations alone (GPT-4V achieves only 42.99% overall accuracy), yet they plan remarkably well once an explicit preference label is provided.
  • Visual Overfitting vs. Symbolic Invariance: In generalization tests across different rooms and objects, vision-based models suffer significant performance drops because they overfit to superficial visual layouts. In contrast, symbol-based LLMs generalize robustly.

Key Results

  • Planning Accuracy (Levenshtein Distance):
  • End-to-end: Video-based models averaged Levenshtein distances of 22.99–25.35 tokens (against ground-truth lengths of ~15.8 at option level and ~35.9 at sequence level).
  • Two-stage: Decoupling inference and planning lowered overall edit distance significantly (GPT-4V dropped from 24.69 to 17.94; GPT-4.1 dropped from 16.95 to 8.22). When supplied with ground-truth preferences, GPT-4.1 achieved a near-perfect distance of 5.23.
  • Preference Prediction Accuracy:
  • LLMs operating over symbolic action logs led the pack (GPT-4.1: 79.60%, DeepSeek-R1: 78.62%, Llama-3-8B: 70.08%).
  • Multimodal video models lagged behind (GPT-4V: 42.99%, EILEV: 35.51%, LLaVA-NeXT: 30.86%, ViViT: 6.81%).
  • Few-Shot Scaling: Increasing demonstration quantity from 1 to 3 steadily improved preference identification and downstream execution, though performance saturated around 5 demonstrations.

Limitations and Future Work

  • Synthetic Domain Gap: All demonstrations and scenes are rendered synthetically in Omniverse/OmniGibson, which lacks real-world messiness, sensor noise, and physics variability.
  • Predefined Categorical Labels: Real human preferences are continuous, compound, and often context-dependent; the current benchmark relies on a discrete set of 290 predefined preference templates.
  • Future Directions: The authors plan to collect natural, real-world few-shot demonstrations using egocentric wearable hardware (e.g., smart glasses) to bridge the sim-to-real gap.

Summary updated

Abstract

Effective integration of AI agents into daily life requires them to understand and adapt to individual human preferences, particularly in assistive roles. Although recent studies on embodied intelligence have advanced significantly, they typically adopt generalized approaches that overlook personalized preferences in planning. Cognitive research has demonstrated that these preferences serve as crucial intermediate representations in human decision-making processes and, though implicitly expressed through minimal demonstrations, can generalize across diverse planning scenarios. To systematically address this gap, we introduce the Preference-based Planning (PBP) benchmark, an embodied benchmark designed to evaluate agents' ability to learn preferences from few demonstrations and adapt their planning strategies accordingly. PBP features hundreds of diverse preferences spanning from atomic actions to complex sequences, enabling comprehensive assessment of preference learning capabilities. Evaluations of SOTA methods reveal that while symbol-based approaches show promise in scalability, significant challenges remain in learning to generate plans that satisfy personalized preferences. Building on these findings, we develop agents that not only learn preferences from few demonstrations but also adapt their planning strategies based on these preferences. Experiments in PBP demonstrate that incorporating learned preferences as intermediate representations significantly improves an agent's ability to construct personalized plans, establishing preference as a valuable abstraction layer for adaptive planning.
BibTeX
@misc{xu2025learning,
  author = {Manjie Xu and Xinyi Yang and Wei Liang and Chi Zhang and Yixin Zhu},
  title = {{Learning to Plan with Personalized Preferences}},
  year = {2025},
  eprint = {2502.00858},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.00858},
}

Loading related content...

Figure